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FISCAL YEAR 2005 ENVIRONMENTAL 
PROTECTION AGENCY BUDGET 


THURSDAY, MARCH 11, 2004 

House of Representatives, 
Subcommittee on Environment, Technology, and 

Standards, 

Committee on Science, 

Washington, DC. 

The Subcommittee met, pursuant to call, at 10:00 a.m., in Room 
2318 of the Rayburn House Office Building, Hon. Vernon J. Ehlers 
[Chairman of the Subcommittee] presiding. 
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HEARING CHARTER 

SUBCOMMITTEE ON ENVIRONMENT, TECHNOLOGY, AND 

STANDARDS 

COMMITTEE ON SCIENCE 
U.S. HOUSE OF REPRESENTATIVES 

Fiscal Year 2005 Environmental 
Protection Agency Budget 

THURSDAY, MARCH 11, 2004 
10:00 A.M.-12:00 P.M. 

2318 RAYBURN HOUSE OFFICE BUILDING 


1. Purpose 

On Thursday, March 11, 2004 at 10:00 a.m. the House Science Subcommittee on 
Environment, Technology, and Standards will hold a hearing to examine the Envi- 
ronmental Protection Agency’s (EPA) fiscal year (FY) 2005 budget request that pro- 
poses steep cuts in the Science to Achieve Results (STAR) grants program. Managed 
by EPA’s Office of Research and Development (ORD), the STAR program supports 
research at colleges and universities on a wide array of environmental science 
issues. 

The hearing will examine why the Administration has proposed a $35 million (or 
35 percent) reduction in the grant program and the consequences of the reduction. 1 
Specifically, the proposed reduction would reduce EPA-sponsored external (also 
called extramural) research on pollution prevention and eliminate STAR funding for 
research in the areas of ecological systems, endocrine disruptors, hazardous sub- 
stances and mercury. The hearing will also examine the extent to which the Office 
of Management and Budget’s (OMB) effort to assess the performance of government 
programs under OMB’s Program Assessment Rating Tool (PART) led the Adminis- 
tration to propose these reductions. 

The Subcommittee plans to explore the following questions: 

1. Why does EPA’s FY05 budget request propose to eliminate funding for EPA’s 
STAR research grants to colleges and universities for research on ecological 
systems, pollution prevention, endocrine disruptors, hazardous substances, 
and mercury? What are the consequences of these reductions? 

2. To what extent did the OMB’s PART review of EPA’s ecological and pollution 
prevention research programs drive these reductions? 

3. How can the performance of environmental research programs best be meas- 
ured? How do OMB’s efforts to assess the performance of EPA’s program dif- 
fer from those of the National Academy of Sciences and EPA’s Science Advi- 
sory Board? 

2. Witnesses 

• Mr. Clay Johnson III, Deputy Director for Management, Office of Manage- 
ment and Budget. 

• Dr. Paul Gilman, Assistant Administrator, Office of Research and Develop- 
ment, Environmental Protection Agency. 

• Mr. Paul Posner, Managing Director, Strategic Issues, U.S. General Ac- 
counting Office (GAO); Managed GAO’s recent report, Performance Budgeting: 
Observations on the Use of OMB’s Program Assessment Rating Tool for the 
Fiscal Year 2004 Budget, GAO-04-174. 

• Dr. Genevieve Matanoski, Professor, Johns Hopkins University, Chair, 
EPA Science Advisory Board’s Review of EPA’s FY 2005 Budget Request; 
former Chair, EPA’s Science Advisory Board (SAB). 


1 The proposed reductions to the STAR program described here and in detail below are com- 
pared to the President’s FY04 budget request. EPA’s final FY04 spending will not be available 
until the agency’s operating plan is completed. 
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• Dr. Costel Denson, Professor, University of Delaware, member of the Na- 
tional Academy of Sciences panel that authored The Measure of STAR: Re- 
view of the U.S. Environmental Protection Agency’s Science to Achieve Results 
(STAR) Research Grants Program (2003). 

3. Brief Overview 

The President’s budget for FY05 proposes to cut Science and Technology (S&T) 
at EPA by $92 million, or about 12 percent. The most significant percentage cut to 
S&T (other than the elimination of earmarks) would reduce funding for EPA’s exter- 
nal grants program by almost 35 percent. 

These proposed cuts to the grant program, known as Science to Achieve Results 
(STAR), are especially noteworthy because in the last decade a number of outside 
experts, including the National Academy of Sciences (NAS), have urged EPA to fund 
research outside of its own laboratories. Most notably, the NAS praised the STAR 
grant programs in its 2003 report, The Measure of STAR: Review of the U.S. Envi- 
ronmental Protection Agency’s Science to Achieve Results (STAR) Research Grants 
Program. The report argues that the external STAR grants are a critical means for 
the agency to have access to expertise that it does not have in-house, and to respond 
quickly to emerging issues. The external grants have also often been favored by crit- 
ics of EPA who view EPA’s in-house scientists as too likely to come up with research 
results that would favor a pre-existing regulatory agenda. 

The cuts to STAR have also attracted attention because they can be seen as a 
test case of how the Administration is using a new system the Office of Manage- 
ment and Budget (OMB) has created to evaluate the effectiveness of federal pro- 
grams and inform spending decisions. The system is known as the Program Assess- 
ment Rating Tool (PART). Each year, OMB is selecting programs at each agency to 
evaluate using PART. It hopes to have reviewed all programs within the next four 
years. 

The proposed cuts to STAR raise two sets of questions about PART: How fairly 
does the PART tool evaluate programs? And how does the Administration apply 
PART evaluations in making budget decisions? The General Accounting Office 
(GAO) recently weighed in on the first question with a report, Performance Budg- 
eting: Observations on the Use of OMB’s Program Assessment Rating Tool for the 
Fiscal Year 2004 Budget (see more detail below). 

The proposed cuts to STAR raise particularly thorny questions about the use of 
PART. Most problematically, STAR grants are part of larger programs targeted at 
particular environmental issues. The STAR grants targeted for cuts are part of five 
separate research programs: ecological systems, pollution prevention, endocrine 
disruptors, mercury, and hazardous substances. The PART was applied to the first 
two programs as a whole; the external grants were not evaluated as a separate ele- 
ment. The other three programs were not evaluated at all. Other issues about the 
way PART relates to the STAR program and EPA’s research and development 
(R&D) programs are described below. 

The Five Programs — What They Do And What Would Be Cut 

Ecological Research. EPA characterizes the $110 million ecological research pro- 
gram as a core or basic research program. Its goal is to develop the scientific under- 
standing to determine ecosystem conditions and trends, diagnose impairments, fore- 
cast ecosystem vulnerability and, ultimately, restore degraded ecosystems. For ex- 
ample, recent STAR funded research has been instrumental in developing scientif- 
ically grounded ways to measure water and ecosystem quality 2 along the Nation’s 
coastal areas and in the mid-Atlantic region. 

According to EPA, the $22 million cut to STAR would eliminate 50 grants in FY05 
across all areas of the ecological research program. The proposed cuts would slow 
research on water quality in FY05 in the Ohio and Mississippi River basin and 
eliminate grants for research in such areas as western rivers and streams, the 
Great Lakes, and the Gulf of Mexico. 

Pollution Prevention. EPA’s pollution prevention research program develops tools, 
technologies, and new systems for preventing pollution. Most of the $50 million pro- 
gram funds applied research, such as the development of innovative technologies for 
reducing the use of hazardous solvents. However, a small portion of the program 
supports extramural grants for more basic research on sustainable technologies. For 
example, in work that EPA carries out in partnership with the National Science 


2 A large portion of the STAR grants in ecological research support the measurement of eco- 
logical conditions and most of ecological condition research is carried out through the Environ- 
mental Monitoring and Assessment program (EMAP) program, a program that the Committee 
has supported in recent years. 
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Foundation, EPA supports research in so-called “green chemistry,” which promotes 
safer chemicals and chemical manufacturing processes. 

The proposed budget would redirect $5 million from research to a pollution pre- 
vention outreach program in another part of the agency. Redirecting these funds 
would eliminate $3 million in STAR funding, which is EPA’s contribution to the 
EPA-NSF partnership. 

Endocrine Disruptors. EPA’s $13 million research program focuses on providing 
a better understanding of the effects of endocrine disruptors and on humans, wild- 
life, and the environment. EPA also is developing new methods to screen chemicals 
for their potential endocrine effects. The program invests in both basic and applied 
research. 

The proposed budget would cut $4.9 million, which would eliminate the entire 
STAR grant research program on endocrine disruptors. The funds would otherwise 
have supported research on the extent to which humans and wildlife are exposed 
to endocrine disruptors, an area that the NAS and the World Health Organization 
have identified as an important research gap. 

Mercury. The goals of EPA’s $7 million mercury research program are to reduce 
and prevent the release of mercury into the environment and to understand how 
mercury moves through the environment. This research supports a variety of the 
agency’s air and water regulatory programs by developing control technologies, 
measuring mercury deposition, and attempting to understand the effect of mercury 
on wildlife. The proposed $1.9 million reduction to the STAR grant portion of the 
mercury research program would eliminate STAR research in FY05 on how and 
where mercury moves through the environment. 

Hazardous Substances. EPA established five university-based centers affiliated 
with 22 universities to address concerns about hazardous substances in the environ- 
ment. Each center has developed a research program to meet regional needs. Cen- 
ters are based at Johns Hopkins University, Louisiana State University at Baton 
Rouge, Purdue University, Colorado State University at Fort Collins and Oregon 
State University at Corvallis. The proposed $2.3 million dollar reduction in STAR 
funding would eliminate ORD’s contribution to these centers. 

What is the Value of Extramural Research at EPA? 

The NAS reviewed EPA’s STAR program in 2003. The report, The Measure of 
STAR: Review of the U.S. Environmental Protection Agency’s Science to Achieve Re- 
sults (STAR) Research Grants Program, reaffirmed earlier National Academy rec- 
ommendations that EPA should maintain an appropriate balance between its intra- 
mural and extramural research programs. (See the Attachment A for a summary 
of the report.) The report gave the STAR program a strong endorsement, calling it 
“EPA’s pre-eminent program that solicits independent scientific and technologic re- 
search from the Nation’s best academic and non-profit research institutions.” It also 
described STAR as enabling EPA to have access to the broad research community, 
fund research at the cutting edge of science, respond quickly to new issues, and ad- 
dress research gaps when EPA lacks the appropriate in-house expertise. It specifi- 
cally praised the unique contributions that the STAR program is making to endo- 
crine disruptors and ecological indicators research. The report concluded by recom- 
mending that ORD maintain STAR funding at a level somewhere between 15 and 
20 percent of ORD’s total budget. 3 

What is the Program Assessment Rating Tool (PART) and what did GAO 
conclude in its recent assessment of the PART? 

The Program Assessment Rating Tool (PART) is a new evaluation tool developed 
by OMB to assess the performance of federal programs and to link that performance 
to spending decisions. It judges programs on their purpose and design, strategic 
planning, management and results, heavily weighting the results portion of the re- 
view. Based on the review, OMB rates a program as either effective, moderately ef- 
fective, adequate, ineffective, or “results not demonstrated.” (See Attachment B for 
a more detailed description of the PART.) OMB plans to apply the tool to all federal 
programs within the next four years. 

The Government Accounting Office (GAO) recently reviewed the PART process, 
though it did not specifically review its application to R&D programs (Performance 
Budgeting: Observations on the Use of OMB’s Program Assessment Rating Tool for 
the Fiscal Year 2004 Budget, GAO-04-174). (See Attachment C for a summary of 


3 If this were the case, and ORD were to be funded at $572, the level requested in the Presi- 
dent’s FY05 budget request, STAR funding would then need to be between $86 million and $114 
million. However, the President’s request for STAR in FY05 appears to be no more than $65 
million, and may be even less than that. 



6 


the GAO report.) According to GAO, the PART process has reinvigorated the Execu- 
tive Branch’s focus on performance budgeting. However, GAO also concluded that 
OMB must do more to ensure fair and consistent application of the PART. GAO de- 
scribed the PART as “a work in progress” and needing “[a]dditional guidance and 
considerable revisions. . .to meet OMB’s goal of an objective, evidence-based assess- 
ment tool.” GAO concluded that there are inherent challenges when applying the 
PART’S restrictive yes/no format to programs with multiple purposes and goals, and 
recommended, among other things, that OMB clarify subjective terminology, provide 
flexibility in judging complex programs, clarify when output and outcome measures 
are appropriate, and increase dialogue with agency staff on such things as the defi- 
nition of the program that will be reviewed. 

The way OMB applied the PART to EPA’s S&T programs may illustrate some of 
GAO’s concerns. For example, many of EPA’s R&D programs have multiple goals 
and purposes, such as combining basic and applied, as well as intramural and extra- 
mural research, in one program. In addition, OMB’s decision to evaluate the overall 
ecological research program may have led to challenges for EPA, which had never 
evaluated the entire program . 4 

What did the PART evaluation conclude about the ecological research and 
pollution prevention programs and was the PART applied fairly? 

OMB used the PART to evaluate two of the five programs with extramural re- 
search elements that were cut in the President’s budget. OMB concluded that the 
ecological and pollution prevention research programs could not “demonstrate re- 
sults,” because neither had adequate standards to measure the progress of the pro- 
grams . 5 

EPA has said it disagrees with how OMB applied the PART and the conclusions 
it reached. The primary area of disagreement appears to be over how to measure 
the performance of EPA’s R&D programs . 6 For example, EPA views the ecological 
research program as more of a basic research program, making it similar to re- 
search supported by the National Science Foundation (NSF). In OMB’s PART review 
of NSF programs, it measured the performance of those programs using process in- 
dicators, such as whether the agency has conducted the appropriate peer reviews 
and how quickly it processes research grants. EPA maintains that its program 
should be similarly evaluated. 

OMB, however, seems to want more than process measures to evaluate EPA’s 
R&D programs. According to both EPA and the public PART review documents, 
OMB seems to want EPA to measure its programs less on process and more on out- 
put and outcome measures, such as the degree to which others used the products 
of the research, and how much pollution the research reduced or might reduce in 
the future. EPA agrees that its research strategic plans should make the connection 
between research and eventual reductions in pollution. However, it does not believe 
that either its basic or applied research programs should be held accountable for the 
actions of others who are outside of EPA’s control and who may or may not use 
EPA’s research products. 

EPA seems to believe that its R&D programs should be evaluated through a peer 
review process that considers the degree to which the research reflects the state of 
the science, adds knowledge to the field, and creates tools and methods that others 
could use. This view seems consistent with the views of EPA’s SAB and various 
NAS reports, such as the STAR report, on the proper way to evaluate R&D. These 
reports have also praised EPA’s for developing high quality basic, extramural re- 
search programs that develop knowledge, but are not tied to regulatory results. 

Why was the ecological research and pollution prevention program cut and 
who decided to cut it? Was the decision related to the PART? 

The Administration’s proposed budget clearly shows that OMB decided to cut the 
ecological and pollution prevention research programs because of their low PART 
scores. OMB’s specific mention of the amount of the proposed reduction in the PART 
summaries makes this readily apparent. However, what is less clear is why these 
programs were cut when many other EPA programs could not demonstrate results 


4 EPA has evaluated subprograms that cut across all of its research programs, such as the 
STAR program, and elements of the ecological research program, such as Environmental Moni- 
toring and Assessment Program. 

5 See OMB’s PART summaries, where it specifies reductions to these programs at http: I / 
www.ivhitehouse.gov / omb I budget / fy2005 / pdf / ap -cd -rom I part.pdf 

6 EPA also disagrees with OMB’s conclusion that the ecological research program failed to co- 
ordinate its research agenda within EPA or with outside agencies or researchers (suggesting 
redundancies with other programs), and that previous evaluations of the ecological research pro- 
gram were too focused on process measures. 
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either. OMB evaluated a total of 20 EPA programs in FY04 and could not determine 
results for 13 of them. Of those 13, some are proposed for decreases, some for in- 
creases and others for flat funding. The rationale for the uneven treatment is un- 
clear. 

Why was the STAR portion of these programs cut and who decided to cut 
it? 

OMB’s PART review did not address the STAR program, other than a brief men- 
tion of NAS’s positive review of the STAR-funded ecological research. As a result, 
it appears that the STAR cuts may have emerged from EPA as it decided how to 
allocate the overall reductions required by OMB. The ultimate reason, however, is 
difficult to know because final decisions on reductions are usually made in negotia- 
tions between the agency and OMB, which are not made public. 

4. Witness Questions 

Questions for Clay Johnson III, Deputy Director for Management, Office of Manage- 
ment and Budget 

In your testimony, please describe the justification for the proposed reductions to 
EPA’s STAR grant research on ecological systems, pollution prevention, endocrine 
disruptors, and mercury. In particular, please focus your testimony on the following 
questions: 

1. To what extent is the proposed reduction in each of these research areas 
based on the evaluation of EPA’s research programs that OMB undertook 
with the PART? 

2. Given that the PART review did not specifically assess the extramural por- 
tion of the research programs, why was the extramural portion of the pro- 
gram cut? 

3. Why does OMB’s PART review of the ecological research program tend to 
treat it as an applied research program when EPA characterizes it more as 
basic research? Is a single performance score under the PART tool appro- 
priate for reviewing a diverse research program, such as EPA’s endocrine 
disruptor research, that combines basic, applied, intramural and extramural 
research? To what extent, should EPA’s R&D performance measures be 
based on specific regulatory program outcomes or environmental outcomes? 

4. Does OMB agree with the Government Accounting Office’s recommendations 
for improving the PART process and its content? How will you implement 
those recommendations? 

Questions for Paul Gilman, Assistant Administrator, Office of Research and Develop- 
ment, U.S. EPA 

Please give a brief description of the STAR extramural research program and how 
it fits into EPA’s overall R&D program. In addition, please answer the following 
questions: 

1. Given the positive review by the National Academy of Sciences of the STAR 
program last year, why was the STAR program cut? 

2. Given the elimination of EPA’s STAR grants for research on ecological sys- 
tems, pollution prevention, endocrine disruptors and mercury, does EPA now 
believe it no longer important to seek out the expertise of university re- 
searchers in these fields. 

3. How does EPA’s ecological research compare to research supported by the 
National Science Foundation and other federal research programs? To what 
extent does EPA coordinate its research with those agencies? 

4. Does EPA characterize its research on ecological systems, pollution preven- 
tion, endocrine disruptors and mercury as basic or applied research? 

5. What performance measures are most appropriate for evaluating these pro- 
grams? To what extent does EPA believe that the performance measures for 
these programs should be tied to the outcomes of specific regulatory pro- 
grams or environmental outcomes? 

6. What research would not be done as a result of the proposed reductions and 
what impact would this have on our scientific understanding and EPA’s reg- 
ulatory programs? 
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Questions for Paul Posner, Managing Director, Strategic Issues, U.S. Government Ac- 
counting Office. 

In your testimony, please describe GAO’s findings and recommendations con- 
cerning the PART and answer the following questions: 

1. What unique problems do research programs raise for evaluation tools like 
PART? What types of evaluation techniques and performance measures are 
most appropriate for reviewing basic research programs and what types are 
most appropriate for applied research? How should OMB decide whether a 
research program should be evaluated as a basic or applied program? 

2. How should the PART deal with programs that have several distinct ele- 
ments, for instance a single research program that funds both basic and ap- 
plied, and intramural and extramural research? 

Questions for Gene Matanoski, Professor, Department of Epidemiology, Johns Hop- 
kins University; Chair, EPA Science Advisory Board FY05 Budget; Former 
Chair, EPA Science Advisory Board 

In your testimony, please describe the Science Advisory Board’s views on the pro- 
posed cuts to the STAR grant program and OMB’s PART review of the ecological 
and pollution prevention research programs. In addition, please answer the fol- 
lowing questions: 

1. What has the SAB recommended to EPA in terms of balancing its research 
and development (R&D) investments between intramural and extramural re- 
search and between basic and applied research? Are SAB’s recommendations 
consistent with recommendations from other reviews of EPA’s science pro- 
grams? 

2. What performance measures are most appropriate for evaluating EPA’s re- 
search programs on ecological systems, pollution prevention, endocrine 
disruptors and mercury? To what extent should performance measures differ 
for basic and applied research programs at EPA? Should EPA’s R&D per- 
formance measures be tied to the outcomes of specific regulatory programs 
or environmental outcomes? 

Questions for Costel Denson, Professor, Department of Engineering, University of 
Delaware; Member of the National Research Council panel for the report, The 
Measure of STAR 

1. How important are the extramural portions of EPA’s research efforts, includ- 
ing those for research on ecological systems, pollution prevention, endocrine 
disruptors, and mercury? What are the likely effects the elimination of these 
grants will have on our scientific understanding and EPA’s regulatory pro- 
grams? 

2. How important is it that EPA ensures that some portion of its environmental 
research funding support extramural research? What portion is an optimal 
amount? 

3. What performance measures are most appropriate for judging the perform- 
ance of EPA’s STAR grant program? To what extent should the STAR pro- 
gram be evaluated as basic research or applied? How well does the STAR 
program perform relative to other federal research programs of similar de- 
sign? 

4. What actions should EPA take to strengthen its STAR research grant pro- 
gram? 

5. Attachments: 

Attachment A. Summary of the NAS Report on STAR 
Attachment B. Summary of the PART program and process. 

Attachment C. Summary of the GAO Report on PART 
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Attachment A 


THE MEASURE OF 

STAR 

Review of the U S. Environmental Protection 
Agency’s Science to Achieve Results (STAR) 
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Summary 


In an effort to improve the scientific foundation of the Environmental Pro- 
tection Agency (EPA) decision-making process, EPA’s Office of Research 
and Development (ORD) created the Science To Achieve Results (STAR) 
program in 1995. The STAR program is a competitive, peer-reviewed, 
extramural research grants program intended to increase the agency’s access 
to the nation’s best scientists and engineers in academic and other nonprofit 
research institutions. It supports research pertaining to human health and 
the environment and is designed to maximize the independence of the re- 
searchers supported and to provide an equal opportunity for all researchers 
to qualify for support. 

The STAR program, currently managed by ORD’s National Center for 
Environmental Research (NCER), is integrated into EPA’s overall research 
program through extensive planning and consultation with the agency’s 
other research centers and laboratories and its program and regional offices. 
The research sponsored by the STAR program allows the agency to fill 
information gaps that are not addressed completely by its intramural re- 
search program and to respond to new issues that the EPA laboratories are 
not able to address. 

The research support awarded by the STAR program is of three main 
kinds: grants awarded to individual investigators or small groups of investi- 
gators, grants awarded to multidisciplinary (and sometimes multi-institu- 
tional) research centers, and fellowships to support graduate work (at the 
master’s and Ph.D. levels) in environmental sciences. The program has 
been funded at about $100 million per year over the last few years and 
accounts for 15-20% of ORD’s research budget. The program has lever- 
aged its funds by forming partnerships with other agencies that support 


1 
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similar kinds of research. Since it was established, the components and 
management of the program have adapted in response to changing agency 
needs, experience gained in operating the program, and external reviews. 


CHARGE TO THE COMMITTEE 

In 2000, EPA asked the National Research Council to conduct an inde- 
pendent assessment of the STAR program. In response, the Research Coun- 
cil established the Committee to Review EPA’s Research Grants Program, 
which prepared this report. The committee was given the following task: 

The NRC committee will conduct a program review of EPA’s Sci- 
ence To Achieve Results (STAR) competitive extramural research 
grants program. Using information to be obtained from EPA, 
STAR grant recipients, and other sources, the committee will assess 
the program’s scientific merit, its demonstrated or potential impact 
on the agency’s policies and decisions, and other program benefits 
that arc relevant to EPA’s mission. The committee will recom- 
mend ways to enhance the program’s scientific merit, impact, and 
other benefits. In the context of other relevant research conducted 
or funded by EPA, and in comparison with other basic and applied 
research grant programs, this assessment will address the STAR 
program’s research priorities, research solicitations, peer-review 
process, ongoing research projects, results and dissemination of 
completed research, and other aspects to be identified by the com- 
mittee. 

In undertaking its review, the committee held three public sessions in 
which it heard presentations about the STAR program by EPA officials and 
others. The EPA officials represented NCER and other EPA research and 
program offices. The public sessions included presentations by representa- 
tives of other federal agencies that support extramural research and by 
experts in evaluating research programs. Committee members also inter- 
viewed STAR project officers and STAR grant and fellowship recipients 
and attended STAR sponsored workshops and meetings. NCER staff pro- 
vided the committee with substantial amounts of information regarding the 
operation and financing of the program. 
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THE COMMITTEE’S EVALUATION 

The committee’s evaluation of the STAR program focused on the pro- 
gram’s quality, relevance, and performance as described in the recent Office 
of Management and Budget (OMB) guidelines on evaluating research pro- 
grams. The committee used metrics that grew out of its review of informa- 
tion available from EPA and of metrics used by EPA and other organiza- 
tions. The metrics, which are both quantitative and qualitative, assisted the 
committee in forming judgments regarding the scientific merit of the pro- 
gram and its impact on the agency. 

The committee recognizes that the STAR program is still too young to 
provide all the information needed for a full evaluation of the extent, im- 
pact, and value of its activities. Evaluation of research results is difficult 
and requires substantial elapsed time; for a given topic, it can take 3-5 years 
from the initiation of laboratory or field expenments to the analysis and 
publication of results. Considerably more time must elapse to realize the 
impact of published research on the scientific and regulatory communities. 
Nevertheless, the committee judged that it had sufficient information to 
evaluate how the STAR program operates and its value to the nation’s 
overall environmental research and management efforts. 

To effectively communicate its findings in this summary, the committee 
developed and addressed a series of specific questions that it believed 
would be of greatest interest to the audience of this report. On the basis of 
its evaluation, the committee unanimously arrived at the following conclu- 
sions and recommendations. 


Should the STAR program continue to be part of the ORD research 
program? 

Finding. EPA requires a strong and balanced science and technology 
research program to fulfill its mission properly. The STAR program is an 
important part of the overall EPA research program. 

Several previous reports by EPA and the National Academies have 
addressed the question of whether EPA should have its own research pro- 
gram or rely on research results developed elsewhere. Those reports all 
concluded emphatically that EPA needs its own strong research program to 
meet the needs of its mission. The committee concurs with that conclusion. 
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The STAR program is EPA’s preeminent program that solicits inde- 
pendent scientific and technologic research from the nation’s best academic 
and other nonprofit research institutions. The program has established and 
maintains a high degree of scientific excellence. By funding the majority 
of its research efforts through broadly advertised, competitive grants, the 
STAR program provides the agency access to independent information, 
analyses, and perspectives. 

The research portfolio of the STAR program is derived directly from 
the strategic plans of EPA and ORD and from ORD’s more-detailed re- 
search strategies that address particular topics. It is an integrated part of 
EPA’s research program. The STAR program provides the agency access 
to a broad community of researchers, allows it to fund research at the cut- 
ting edge of science, and assists it in addressing information gaps that it 
does not have the internal resources to address properly. The STAR pro- 
gram also encourages its grantees to disseminate their research results 
widely to promote their rapid and widespread use. 

For all those reasons, STAR research effectively expands the nation’s 
scientific foundation for protecting human health and the environment. 
Moreover, by expanding environmental research and analysis capabilities 
in many of the nation’s academic and other nonprofit research institutions 
and by supporting young scientists interested in environmental research, the 
STAR program actively expands the nation’s environmental-science infra- 
structure. 

Recommendation. The STAR program should continue to be an im- 
portant part of EPA’s research program. 


What is the unique contribution of the STAR program? 

Finding. The STAR program funds important research that is not 
conducted or funded by other agencies. The ST AR program has also made 
commendable efforts to leverage funds through establishment of research 
partnerships with other agencies and organizations. 

The STAR program provides EPA with access to independent research 
that is directly relevant to its mission. The program makes strong efforts to 
ensure that the results of its research are expeditiously communicated to 
relevant EPA program offices and to other potential users. The STAR pro- 
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gram gives primary potential users of research results a unique role in help- 
ing to plan the research and to identify the specific high-quality proposals 
that will be of greatest value to them. The exploratory and core research 
that the program sponsors alerts the agency to possible emerging issues, 
providing more opportunity for the agency to consider how it might best 
address them. 

Much of the research funded by STAR would not have been undertaken 
without the program, because it is not conducted or funded by other agen- 
cies. For instance, EPA is one of the few agencies that provide extramural 
funding for examining the impacts of endocrine disruptors on ecosystem 
processes. The STAR ecologic-indicators program is the primary source of 
support of research on the development of water-quality indicators for 
biologic monitoring. The interdisciplinary centers that ST AR has supported 
also represent an innovative approach to supporting research that will be 
most relevant for environmental decision making in several important top- 
ics. 

Finally, the STAR program has been successful in working with other 
agencies that have similar or complementary research interests through 
research partnerships and in obtaining supplementary funding. That not 
only leverages additional funds for research projects of interest to STAR but 
also helps to increase the partner agencies ’ awareness of the pertinent issues 
and information needs of EPA. The STAR program’s ability to establish 
partnerships has increased as more funds have been allocated to it. 

Recommendation. STAR should continue to partner with other gov- 
ernment and nongovernment organizations to support research of mutual 
interest and of relevance to EPA’s mission, explore innovative approaches 
for carrying out this research, and sponsor a diverse portfolio of research 
that alerts the agency to emerging issues and provides independent analyses 
of issues that the agency is currently addressing. 


Does the STAR program have adequate processes to ensure that it is 
sponsoring high-quality and relevant research? 

Finding. The procedures that STAR has established for soliciting and 
selecting the highest-quality research proposals compare favorably with the 
procedures established by other research agencies. STAR’S procedures for 
incorporating mission relevance into its research-planning process and in 
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the selection of proposals to fund exceed those practiced by most other 
agencies. 

The ST AR program has developed a grant-award process that compares 
favorably with and in some ways exceeds that in place at other agencies that 
have extramural research programs, such as the National Science Founda- 
tion (NSF) and the National Institute of Environmental Health Sciences. 
An unusually high degree of planning goes into identifying the specific 
research subjects to be supported. The agency also puts considerable time 
and thought into preparing effective research solicitations and into funding 
projects that are relevant to its mission and program needs. 

EPA spends substantial effort in defining its research agenda, and the 
STAR program submits its proposed requests for applications (RFAs) to 
extensive review within the agency. Those efforts are intended to ensure 
that the research requests are focused on the issues most important to EPA. 

However, the STAR program makes insufficient use of outside experts 
in planning its research agenda and in identifying the most important gaps 
in scientific knowledge. As a result, some of its early RFAs were not as 
well focused as they should have been. 

In soliciting research proposals, STAR makes a substantial effort to 
reach out to the broad scientific community and to attract the most capable 
scientists. The RFAs are distributed widely through EPA’s Web site, the 
Federal Register, announcements at professional meetings, and e-mail 
distributions to individuals or institutions that sign up on the STAR Web 
site. When the desired research is outside EPA’s traditional research fields 
and might therefore include scientists not already involved with the 
agency’s research program, STAR often solicits the help of other agencies 
that traditionally work with these scientists to ensure that they are aware of 
the funding opportunities. 

The STAR program has established a rigorous peer-review process. 
Such peer-review processes are a key part of the foundation on which excel- 
lence is achieved in all research programs, including those of the National 
Institutes of Health (NIH) and NSF. The agency has taken effective steps 
to ensure that the process does not suffer from conflicts of interest and is 
independent. EPA provides a “firewall” that shields the peer-review pro- 
cess from the influence of the project officers and staff who oversee the 
individual-investigator, fellowship, and center awards. 

Recommendation. The STAR program should continue to improve the 
focus of its RFAs, and when the agency does not have the capacity inter- 
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nally to adequately define the state of the science in a particular research 
field, STAR should consider greater use of external experts to assist in 
identifying the highest-priority research and data gaps. 


Is the STAR program producing high-quality research results? 

Finding. Although it is still too early for comprehensive evaluations 
of the research results of the STAR program, some STAR research efforts 
have already substantially improved the scientific foundation for decision 
making, and the results produced by STAR investigators have been widely 
published in peer-reviewed journals. 

Evaluating the quality of research results is difficult and necessarily 
involves substantial judgment on the part of scientists with expertise in the 
research fields being reviewed. In addition, because of the relative youth 
of the STAR program, only about 40% of STAR research projects funded 
to date have been completed. 

However, many STAR projects have resulted in articles in highly re- 
spected, peer-reviewed journals — a traditional measure of research quality. 
These STAR research results have already helped to improve our under- 
standing of the causes, exposures, and effects of environmental pollu- 
tion — information critical to improving the scientific foundation for deci- 
sion making. For instance, STAR-funded research on particulate matter has 
helped to improve our understanding of the biologic mechanisms by which 
inhaled ambient particles cause health effects and the nature of some of 
those effects. These data are cntical to future regulatory decisions regard- 
ing our nation’s ambient air quality. 

A limited bibliometric analysis by the committee indicated that the 
citation rate of STAR-supported research is comparable with that of re- 
search in the same fields funded by other research organizations and under- 
taken by other investigators. For instance, in 1997, the average number of 
citations of STAR-funded ecologic research was 10.5, compared with 10.3 
citations of the work of all other investigators in ecology. 

The committee also reviewed the backgrounds and accomplishments of 
a sample of STAR-funded principal investigators. The review indicated 
that the STAR program was funding many scientists with outstanding cre- 
dentials; they have impressive research track records and are leaders in their 
fields; are editors of journals or officers in societies and have received 
awards of distinction; and were attracted to the STAR program from fields 
outside EPA’s mission. 
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On the basis of the STAR program’s process for awarding grants, the 
quality of the individuals and institutions funded by the program, and the 
highly competitive nature of the awards, the committee is confident that the 
products of STAR grants are of the highest quality. 

Recommendation. EPA should continue its efforts to attract “the best 
and the brightest” researchers to compete for STAR funding. 


Are the STAR program results useful for EPA decisions and processes? 

Finding. The STAR portfolio effectively supports EPA’s mission, 
Government Performance and Results Act goals, and ORD strategic plans. 
Specific STAR research projects have yielded significant new findings and 
knowledge critical for regulatory decision making. 

The STAR program is too young to be able to document fully the extent 
to which its research results are being used to inform development of new 
regulations and environmental-management decisions. Even with respect 
to projects that have been completed, there is often a substantial delay be- 
tween when the research results are produced and the agency decides to 
undertake rule-making or other actions to address the issues that were stud- 
ied. 

However, some ST AR projects have already yielded information impor- 
tant for environmental decision making. For example, STAR-sponsored 
research in endocrine disruptors, particulate matter, and ecologic assessment 
has resulted in groups of peer-reviewed publications of immediate use in 
understanding causes, exposures, and effects of environmental pollution. 
Those results are directly relevant to EPA’s mission to “protect human 
health and to safeguard the natural environment — air, water, and 
land — upon which life depends.” For instance, STAR-funded research on 
particulate matter has helped to improve our understanding of the biologic 
mechanisms by which inhaled ambient particles cause health effects. Re- 
search on ecologic indicators has led to the development of a dynamic, 
economically linked model to evaluate the driving forces and ecologic 
consequences of land-use change. 

In research fields in which EPA does not already have substantial ex- 
pertise, the committee suggests that the program consider bringing in out- 
side experts to assist in assessing the state of the science while the research 
program is being planned and then to synthesize the contributions of the 
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STAR-supported research when it has been completed. Such assessments 
would help EPA to target RFAs and then analyze the utility of the com- 
pleted research in providing critical knowledge or otherwise strengthening 
and improving the foundation for environmental decision making. 

To ensure the usefulness of STAR research results, it is also important 
for the ST AR program to maintain a balanced research portfol io, including 
balances between “core” and “problem-driven” research and between hu- 
man health and ecologic research. 

Recommendation. The STAR program and ORD should develop 
mechanisms for documenting the extent to which its research is being used 
to support the agency’s environmental decision making, should consider 
using outside experts to help document systematically the “state of the sci- 
ence” before research is initiated, and should synthesize the results of the 
research when it is completed to identify the specific contributions that 
STAR and ORD research has made to providing critical information. 


Is the STAR program effective in providing results relevant to the 
appropriate audiences? 

Finding. The STAR program has been commendably aggressive in 
experimenting with innovative approaches to communicating the results of 
its funded research to a wide variety of users and audiences, but its success 
in these efforts has been uneven. 

The STAR program supports research of potential value to a variety of 
users and audiences, both in and outside EPA. Much of the research is 
aimed primarily at the scientific community and those responsible for pro- 
viding technical support for environmental-management decisions. For the 
scientific community, the primary communication product is peer-reviewed 
journal articles, and the program has been successful in encouraging the 
preparation of these articles. 

The program, however, also has other potential users, at least for the 
results of some of its research. They include other federal agencies; indus- 
try; state, tribal, and local governments; nonprofit environmental organiza- 
tions; and international environmental agencies. The audience for some 
projects appears to be local communities (for instance, communities that 
have received Environmental Monitoring for Public Access and Community 
Tracking, or EMPACT, grants) or the general public; disseminating results 
to such audiences is much more difficult. 
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The STAR program has experimented aggressively with a wide variety 
of communication mechanisms. Information is available to the public on 
EPA’s Web site concerning every step of the STAR process, from the initial 
solicitation of research proposals, through the award of grants, to the final 
research results. STAR researchers are required to prepare annual progress 
reports, which are made available to the public in summary form. The 
STAR program also produces several series of reports that summarize re- 
search results in selected topics. In all those efforts, the program appears 
to substantially exceed the dissemination efforts of most other research- 
sponsoring organizations, both in and outside the federal government. 

Nevertheless, the STAR program could substantially improve its dis- 
semination efforts by directing its communication efforts more effectively 
to specific users and audiences. The program does not always clearly iden- 
tify the users and audiences for its research results. Often, the research 
results are produced, and then EPA assesses how to communicate them. 
The dissemination process would be much more effective and efficient if 
the potential audiences were clearly identified before the research began and 
if the focus were maintained throughout the research process and the prepa- 
ration of reports. 

In some cases, the effective dissemination of results should be primarily 
STAR’S responsibility. In other cases, STAR’S contributions will be a 
component of a larger research effort, and the primary dissemination re- 
sponsibility should lie with ORD or EPA. In all cases, however, dissemina- 
tion efforts are likely to be more effective if the intended audiences are 
clearly defined from the beginning of the STAR grants process. 

Recommendation. The STAR program should clearly identify the 
intended audiences for proposed research results as early in the process as 
possible and should identify the audiences in RFAs. When appropriate, 
EPA should consider involving representatives of the intended audiences 
from outside the agency in helping to define the relevant research results 
and the strategy for their dissemination. 


Should the fellowship program continue to be part of the ORD research 
program? 

Finding. The STAR fellowship program is a valuable mechanism for 
enabling a continuing supply of graduate students in environmental sciences 
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and engineering to help build a stronger scientific foundation for the na- 
tion’s environmental research and management efforts. 

The fellowship program was established to “encourage promising stu- 
dents to obtain advanced degrees and pursue careers in environmentally 
related fields” and to develop the next generation of environmental scien- 
tists. It is the only federal fellowship program exclusively designed for 
students pursuing advanced degrees in environmental sciences. It has 
achieved its goals, as evidenced by the extraordinary competition for the 
fellowships and the rigorous, independent selection process. Of the fellow- 
ship applications that STAR receives annually, only 125 fellowships, or 
1 0% of all applicants, receive funding. Of the more than 1 00 former EPA 
fellowship recipients that were contacted by the committee, over 95% indi- 
cated high satisfaction with the program, and nearly 90% have remained in 
the environmental field, thus successfully contributing to the long-term 
program goals. 

Recommendation. Given the nation’s continuing need for highly 
qualified scientists and engineers in environmental research and manage- 
ment, the STAR fellowship program should be continued and funded. 


Are the STAR program ’s funds adequate to achieve its objectives? 

Finding. STAR is only able to fund less than 15% of the proposals 
received for its individual investigator and center grants, and its funding has 
not kept pace with the rate of inflation. 

NIH and NSF strive to fund, on the average, 25-30% of the proposals 
received. STAR’S budget allows it to fund only 10-15% of the proposals 
it receives and only about 60% of those rated “excellent” or “very good” by 
its independent quality peer-review panels. By that measure, STAR does 
not have sufficient funds to recognize all the best proposals received. 

To be effective in its partnerships with other agencies, STAR must have 
sufficient funding to allocate to subjects of mutual interest to make it worth 
the extra administrative effort that partnerships require. The partnerships 
benefit STAR as a result of both the funds they leverage and the reputation 
they bring to the program. 

Although the STAR program’s budget grew rapidly in its first 3 years, 
it has not kept pace with general inflation in the last few years. That is 
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particularly true of the STAR fellowship program. The effect of that bud- 
getary situation is exacerbated by the fact that costs of research have out- 
paced general inflation for more than a decade. Therefore, at present, ST AR 
funds buy less research than the same amount of money could have bought 
several years ago. 

It is appropriate to consider the funding of the STAR program in the 
context of the overall funding for all of ORD, which also has not kept pace 
with inflation. STAR currently represents about 1 8% of ORD’s total fund- 
ing. The committee considers that percentage to be a reasonable recogni- 
tion of the value of independent peer-reviewed research to the agency. 

Recommendation. STAR program funding should be maintained at 
15-20% of the overall ORD budget, even in budget-constrained times. 
However, budget planners should clearly recognize the constraints of not 
having inflation escalators to maintain the level of effort of the entire pro- 
gram. 


How should the STAR program be evaluated? 

Finding. There are no easy answers when it comes to identifying met- 
rics for evaluating research programs, and the best approach for evaluating 
the STAR program is to establish a structured system of reviews by panels 
of experts. 

The STAR program has undergone a substantial — some might say 
excessive — number of reviews. Most of the reviews have focused on the 
program’s procedures; it is too early in the program’s life to be able to 
evaluate the research products fully. Too many reviews can be disruptive 
to the program and can divert the program’s attention and resources from 
its primary purpose. 

The committee, in its own evaluation of STAR, assessed the quality, 
relevance, and performance of the program, as set forth in recent OMB 
research and development criteria, by using qualitative and quantitative 
metrics selected on the basis of its review of information available from 
EPA and metrics used by EPA and other organizations. That is one ap- 
proach for reviewing the STAR program and similar programs. Several 
examples of qualitative and quantitative metrics that were used for evaluat- 
ing the STAR program are these: Does the STAR program have a clearly 
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defined plan for regular, external reviews of its research quality, and has 
this plan been effectively carried out? Has the program made significant 
contributions to advancing the state of the science in particular research 
topics? Docs the program award grants expeditiously? Does the program 
have a schedule for the products it intends to produce and how well is it 
adhering to the schedule? 

The committee’s judgment is that quantitative metrics, although out- 
wardly simpler to use, are not necessarily more informative than qualitative 
metrics. In some cases, quantitative metrics can be misleading, and empha- 
sizing inappropriate metrics can distort the research outputs of a program. 
Qualitative metrics are less likely to have such effects, but they need to be 
interpreted carefully. 

The committee judges that expert review by a group of people with 
appropriate expertise is the best method of evaluating broad research pro- 
grams, such as the STAR program. Expert review is appropriate for evalu- 
ating both the processes and the products of the STAR program. The types 
of experts needed depend on the level of review being conducted — indi- 
vidual projects or programmatic levels. Both qualitative and quantitative 
metrics can provide valuable support for such expert reviews. 

In planning for future reviews, the committee recommends that STAR 
and ORD consider an evaluation structure for the STAR program that has 
four levels: level 1 should examine the individual research projects, level 
2 should focus on topics or groups of research projects on the same subject, 
level 3 should address the STAR program as a whole, and level 4 should 
tackle the question of how the STAR program relates to the broader institu- 
tions of ORD and EPA. The primary mechanism of review at levels 2-4 
should be the panel of independent experts with the appropriate scientific, 
management, and policy backgrounds; the panels’ evaluations can use 
metrics appropriate to the specific level of review. Such a structured review 
strategy could replace the number of ad hoc, unplanned, and uncoordinated 
reviews. 

Recommendation. STAR and ORD should establish a structured 
program of reviews by panels of independent experts and should collect the 
appropriate information to support these reviews. 
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Summary of the PART program and process 

What is the Program Assessment Rating Tool (PART)? 

The PART is the latest executive branch initiative designed to 
better align spending decisions and program performance, often 
called “performance budgeting. The current statutory framework 
for performance budgeting is the Government Performance and Re- 
sults Act, which became law in 1993. President George W. Bush 
announced his strong support for linking performance and budget, 
when he made it a major goal of the “President’s Management 
Agenda.” A key element in accomplishing this objective is the Of- 
fice of Management and Budget’s PART. The Administration views 
the PART as enabling more effective implementation of GPRA, and 
as aligned more closely with budgeting decisions than GPRA. 

According to OMB, the PART is a diagnostic tool meant to pro- 
vide a consistent approach to evaluating federal programs 7 as part 
of the budget formulation process. The PART, which is imple- 
mented by OMB’s budget examiners, requires an examiner to an- 
swer 25 yes/no questions under four overarching categories. Each 
of the four categories is given a specific weight for determining an 
overall numerical score for each program. The categories and their 
weightings are: (1) program purpose and design (e.g., is the design 
clear and purpose sensible?) (20 percent); (2) strategic planning, 
(e.g., has the program set appropriate annual and long-term goals?) 
(10 percent); (3) program management, (is there sound financial 
and management oversight?) (20 percent); and (4) program results, 
(has the program is met its annual and long-term goals?) (50 per- 
cent). OMB asks a few supplementary questions, which vary de- 
pending on what type of program is under review, such as block 
grant, regulation, or R&D. Based on the total score, programs are 
rated as either effective, moderately effective, adequate, ineffective, 
or results not demonstrated. 

OMB applied the PART to 234 programs in FY 2004 across all 
federal agencies, and plans to rate nearly 100 percent of all re- 
maining programs over the next four years. Of the 234 programs, 
over 100 programs received ratings “results not demonstrated.” Ac- 
cording to a recent GAO study, discretionary programs that re- 
ceived effective scores tended to see budget increases and programs 
that received ineffective scores tended to receive budget decreases. 
For programs that received “results not demonstrated” scores, the 
budget story was more mixed. According to GAO, programs that re- 
ceived this score tended to indicate programs for which OMB and 
the Agency could not agree on appropriate performance measures. 


7 There is no standard definition of program, though it is intended to capture a set of activities 
clearly recognized as a program, having a discrete budget, or related to the level at which budg- 
et decisions are made. 
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Highlights of GAO-04-1 74, a report to 
congressional requesters 


Why GAO Did This Study 

The Office of Management and 
Budget’s (OMB) Program 
Assessment Rating Tool (PART) is 
meant to provide a consistent 
approach to evaluating federal 
programs during budget 
formulation. To better understand 
its potential, congressional 
requesters asked GAO to examine 
(1) how PART changed OMB’s 
fiscal year 2004 budget decision- 
making process, (2) PART’S 
relationship to the Government 
Performance and Results Act of 
1993 (GPRA), and (3) PART’S 
strengths and weaknesses as an 
evaluation tool. 


What GAO Recommends 


GAO recommends that OMB 
(1) address the capacity demands 
of PART, (2) strengthen PART 
guidance, (3) address evaluation 
information availability and scope 
issues, (4) focus program selection 
on crosscutting comparisons and 
critical operations, (5) broaden the 
dialogue with congressional 
stakeholders, and 
(6) articulate and implement a 
complementary relationship 
between PART and GPRA. 

OMB generally agreed with our 
findings, conclusions, and 
recommendations and stated that it 
is already taking actions to address 
many of our recommendations. 

GAO also suggests that Congress 
consider the need for a structured 
approach to articulating its 
perspective and oversight agenda 
on performance goals and priorities 
for key programs. 

www . gao. gov/cgi-bin/get rpt ?G AO-04- 1 74 . 

To view the full product, including the scope 
and methodology, click on the link above. 

For more information, contact Paul Posner at 
(202) 512-9573 or posnerp@gao.gov. 
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What GAO Found 

PART helped structure OMB’s use of performance information for its 
internal program and budget analysis, made the use of this information more 
transparent, and stimulated agency interest in budget and performance 
integration. OMB and agency staff said this helped OMB staff with varying 
levels of experience focus on similar issues. 

Our analysis confirmed that one of PART’S major impacts was its ability to 
highlight OMB’s recommended changes in program management and design. 
Much of PART’S potential value lies in the related program 
recommendations, but realizing these benefits requires sustained attention 
to implementation and oversight to determine if desired results are achieved. 
OMB needs to be cognizant of this as it considers capacity and workload 
issues in PART. 

There are inherent challenges in assigning a single rating to programs having 
multiple purposes and goals. OMB devoted considerable effort to promoting 
consistent ratings, but challenges remain in addressing inconsistencies 
among OMB staff, such as interpreting PART guidance and defining 
acceptable measures. Limited credible evidence on results also constrained 
OMB’s ability to rate program effectiveness, as evidenced by the almost 50 
percent of programs rated “results not demonstrated.” 

PART is not well integrated with GPRA — the current statutory framework 
for strategic planning and reporting. By using the PART process to review 
and sometimes replace GPRA goals and measures, OMB is substituting its 
judgment for a wide range of stakeholder interests. The PART/GPRA tension 
was further highlighted by challenges in defining a unit of analysis useful for 
both program-level budget analysis and agency planning purposes. Although 
PART can stimulate discussion on program-specific measurement issues, it 
cannot substitute for GPRA’s focus on thematic goals and department- and 
govemmentwide crosscutting comparisons. Moreover, PART does not 
currently evaluate similar programs together to facilitate trade-offs or make 
relative comparisons. 

PART clearly must serve the President’s interests. However, the many actors 
whose input is critical to decisions will not likely use performance 
information unless they feel it is credible and reflects a consensus on goals. 

It will be important for OMB to discuss timely with Congress the focus of 
PART assessments and clarify the results and limitations of PART and the 
underlying performance information. A more systematic congressional 
approach to providing its perspective on performance issues and goals could 
facilitate OMB’s understanding of congressional priorities and thus increase 
PART’S usefulness in budget deliberations. 
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The PART has helped to structure and discipline OMB’s use of performance 
information for its internal program analysis and budget review, made the 
use of this information more transparent, and stimulated agency interest in 
budget and performance integration. Both OMB and agency staff noted that 
this helped ensure that OMB staff with varying levels of experience focused 
on the same issues, fostering a more disciplined approach to discussing 
program performance with agencies. Several agency officials also told us 
that the PART was a catalyst for bringing agency budget, planning, and 
program staff together since none could fully respond to the PART 
questionnaire alone. 

Our analysis confirmed that one of the PART’S major impacts was its ability 
to highlight OMB’s recommended changes in program management and 
design. Over 80 percent of the recommendations made for the 234 
programs assessed for the fiscal year 2004 budget process were for 
improvements in program design, assessment, and program management; 
less than 20 percent were related to funding issues. As OMB and others 


Page 4 


GAO-04-174 Performance Budgeting 



27 


recognize, performance is not the only factor in funding decisions. 
Determining priorities — including funding priorities — is a function of 
competing values and interests. Although OMB generally proposed to 
increase funding for programs that received ratings of “effective” or 
“moderately effective” and proposed to cut funding for those programs that 
were rated “ineffective,” our review confirmed OMB’s statements that 
funding decisions were not applied mechanistically. That is, for some 
programs rated “effective” or “moderately effective” OMB recommended 
funding decreases, while for several programs judged to be “ineffective” 
OMB recommended additional funding in the President’s budget request 
with which to implement changes. 

Much of the potential value of the PART lies in the related program 
recommendations and associated improvements, but realization of these 
benefits will require sustained attention to implementation and oversight in 
order to determine if the desired results are being achieved. Such attention 
and oversight takes time, and OMB needs to be cognizant of this as it 
considers the capacity and workload issues in the PART. Currently OMB 
plans to assess an additional 20 percent of all federal programs annually. 
Each year, the number of recommendations from previous years’ 
evaluations will grow — and a system for monitoring their implementation 
will become more critical. OMB encouraged its Resource Management 
Offices (RMO) to consider many factors in selecting programs for the fiscal 
year 2004 PART assessments, such as continuing presidential initiatives 
and programs up for reauthorization. While all programs would eventually 
be reviewed over the 5-year period, selecting related programs for review 
in a given year would enable decision makers to analyze the relative 
efficacy of similar programs in meeting common or similar outcomes. We 
recommend that OMB centrally monitor and report on agency 
implementation and progress on PART recommendations to provide a 
govemmentwide picture of progress and a consolidated view of OMB’s 
workload in this area. In addition, to target scarce analytic resources and to 
focus decision makers’ attention on the most pressing policy issues, we 
recommend that OMB reconsider plans for 100 percent coverage of federal 
programs by targeting PART assessments based on such factors as the 
relative priorities, costs, and risks associated with related clusters of 
programs and activities. We further recommend that OMB select for review 
in the same year related or similar programs or activities to facilitate such 
comparisons and trade-offs. 

Developing a credible evidence-based rating tool to provide bottom-line 
ratings for programs was a major impetus in developing the PART. 
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However, inherent challenges exist in assigning a single “rating” to 
programs that often have multiple purposes and goals. Despite the 
considerable time and effort OMB has devoted to promoting consistent 
application of the PART, the tool is a work in progress. Additional guidance 
and considerable revisions are needed to meet OMB’s goal of an objective, 
evidence-based assessment tool. In addition to difficulties with the tool 
itself — such as subjective terminology and a restrictive yes/no format — 
providing flexibility to assess multidimensional programs with multiple 
purposes and goals often implemented through multiple actors has led to a 
reliance on OMB staff judgments to apply general principles to specific 
cases. OMB staff were not fully consistent in interpreting the guidance for 
complex PART questions and in defining acceptable measures. In addition, 
the limited availability of credible evidence on program results also 
constrained OMB staff’s ability to use the PART to rate programs’ 
effectiveness. Almost 50 percent of the 234 programs assessed for fiscal 
year 2004 received a rating of “results not demonstrated” because OMB 
decided that program performance information, performance measures, or 
both were insufficient or inadequate. OMB, recognizing many of the 
limitations with the PART, modified the PART for fiscal year 2005 based on 
lessons learned during the fiscal year 2004 process, but issues remain. We 
therefore recommend that OMB continue to improve the PART guidance by 
(1) clarifying when output versus outcome measures are acceptable and (2) 
better defining an “independent, quality evaluation.” We further 
recommend that OMB both clarify its expectations regarding the nature, 
timing, and amount of evaluation information it wants from agencies for 
the purposes of the PART and consider using internal agency evaluations 
as evidence on a case-by-case basis. 

The PART is not well integrated with GPRA — the current statutory 
framework for strategic planning and reporting. According to OMB 
officials, GPRA plans were organized at too high a level to be meaningful 
for program-level budget decision making. To provide decision makers with 
program-specific, outcome-based performance data useful for executive 
budget formulation, OMB has stated its intention to modify GPRA goals 
and measures with those developed under the PART. As a result, OMB’s 
judgment about appropriate goals and measures is substituted for GPRA 
judgments based on a community of stakeholder interests. Agency officials 
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we spoke with expressed confusion about the relationship between GPRA 
requirements and the PART process. Many view PART’S program-by- 
program focus and the substitution of program measures as detrimental to 
their GPRA planning and reporting processes. OMB’s effort to influence 
program goals is further evident in recent OMB Circular A- 11 guidance 7 
that clearly requires each agency to submit a performance budget for fiscal 
year 2005, which will replace the annual GPRA performance plan. 

The tension between PART and GPRA was further highlighted by the 
challenges in defining a unit of analysis that is useful both for program-level 
budget analysis and agency planning purposes. Although the PART reviews 
indicated to OMB that GPRA measures are often not sufficient to help it 
make judgments about programs, the different units of analysis used in 
these two performance initiatives contributed to this outcome. For the 
PART, OMB created units of analysis that tied to discrete funding levels by 
both disaggregating and aggregating certain programs. In some cases, 
disaggregating programs for the PART reviews ignored the 
interdependency of programs by artificially isolating them from the larger 
contexts in which they operate. Conversely, in other cases in which OMB 
aggregated programs with diverse missions and outcomes for the PART 
reviews, it became difficult to settle on a single measure (or set of 
measures) that accurately captured the multiple missions of these diverse 
components. Both of these “unit of analysis” issues contributed to the lack 
of available planning and performance information. 

Although the PART can stimulate discussion on program-specific 
performance measurement issues, it is not a substitute for GPRA’s 
strategic, longer-term focus on thematic goals and department- and 
govemmentwide crosscutting comparisons. GPRA is a broad legislative 
framework that was designed to be consultative with Congress and other 
stakeholders and allows for varying uses of performance information, 
while the PART applies evaluation information to support decisions and 
program reviews during the executive budget formulation process. 
Moreover, GPRA can anchor the review of programs by providing an 
overall strategic context for programs’ contributions toward agency goals. 
We therefore recommend that OMB seek to achieve the greatest benefit 
from both GPRA and PART by articulating and implementing an integrated, 
complementary relationship between the two. We further recommend that 
OMB continue to improve the PART guidance by expanding the discussion 


7 OMB Circular A-ll, Preparation, Submission, and Execution of the Budget, Section 220. 
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of how programs — also known as “units of analysis” — are determined, 
including recognizing the trade-offs, implications, or both of such 
determinations. 

As part of the President’s budget preparation, the PART clearly must serve 
the President’s interests. However, experience suggests that efforts to 
integrate budget and performance are promoted when Congress and other 
key stakeholders have confidence in the credibility of the analysis and the 
process used. It is unlikely that the broad range of players whose input is 
critical to decisions will use performance information unless they believe it 
is relevant, credible, reliable, and reflective of a consensus about 
performance goals among a community of interested parties. Similarly, the 
measures used to demonstrate progress toward a goal, no matter how 
worthwhile, cannot appear to serve a single set of interests without 
potentially discouraging use of this information by others. We therefore 
recommend that OMB attempt to build on the strengths of GPRA and PART 
by seeking to communicate early in the PART process with congressional 
appropriators and authorizes about what performance issues and 
information are most important to them in evaluating programs. 
Furthermore, while Congress has a number of opportunities to provide its 
perspective on performance issues and goals through its authorization, 
oversight, and appropriations processes, we suggest that Congress 
consider the need for a more structured approach for sharing with the 
executive branch its perspective on governmentwide performance matters, 
including its views on performance goals and outcomes for key programs 
and the oversight agenda. 

In commenting on a draft of this report, OMB generally agreed with our 
findings, conclusions, and recommendations. OMB outlined actions it is 
taking to address many of our recommendations, including refining the 
process for monitoring agencies’ progress in implementing the PART 
recommendations, seeking opportunities for dialogue with Congress on 
agencies’ performance, and continuing to improve executive branch 
implementation of GPRA plans and reports. OMB also suggested some 
technical changes throughout the report that we have incorporated as 
appropriate. OMB’s comments appear in appendix IV. We also received 
technical comments on excerpts of the draft provided to the Departments 
of the Interior, Energy, and Health and Human Services, which are 
incorporated as appropriate. 
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Chairman Ehlers. Good morning. Welcome to today’s hearing on 
the Environmental Protection Agency’s Fiscal Year 2005 Science 
and Technology Budget. I will try to be brief with my opening 
statement so we can get right to the business of the day, and I will 
also note that Congressman Miller is here filling in for Congress- 
man Udall, who is the Ranking Member. 

The role of science in technology at EPA is more important than 
ever. EPA is being asked to address increasingly complex technical 
questions in its regulatory processes and in its search for emerging 
environmental problems. That is why I am surprised and concerned 
at the Administration’s proposal to cut $22 million, which is nearly 
12 percent, from EPA’s science budget. It is hard to understand 
why, even in this tight budget time, the science budget would de- 
serve such a substantial cut. 

Today, the Subcommittee hopes to learn why the Administration 
proposes a cut of $35 million or 35 percent to EPA’s competitive ex- 
ternal research grants to colleges, universities and other research- 
ers, as well as find out the impact of those reductions. This pro- 
posed cut is in the Science to Achieve Results grant program, affec- 
tionately referred to as STAR grant. It would mean less extramural 
research on ecological systems, pollution prevention, endocrine 
disrupters and mercury, among other topics. 

Today, we are also interested in the understanding of the extent 
to which the Office of Management Budget’s effort to assess the 
performance of governing programs under its new Program Assess- 
ment Rating Tool called PART led to these proposed cuts. As a 
strong supporter of linking performance to funding decisions, I look 
forward to a healthy discussion on how we can best evaluate envi- 
ronmental research and developmental programs. The PART is 
playing an increasingly important role in the Administration’s 
budget decisions. It has been applied to hundreds of federal pro- 
grams in the last two years and will be applied to the remainder 
over the next four. With this in mind, it is critically important for 
this Committee and the Congress overall to understand how this 
assessment tool is being used. 

I would also like to mention my concern about the 8.3 million re- 
duction the Administration proposes that would eliminate EPA’s 
research on building decontamination. EPA has a unique role in 
cleaning contaminated buildings and it is not clear why this work 
is being cut, or whether another agency, such as the Department 
of Homeland Security, will pick it up. The full Science Committee 
plans to address this question further at our joint hearing with a 
select Committee on Homeland Security on March 25. 

We have an excellent panel of witnesses today, and I expect that 
we will learn a great deal. I am looking forward to the testimony 
that we will hear. 

The Chair now recognizes Congressman Brad Miller from North 
Carolina for an opening statement. 

[The prepared statement of Mr. Ehlers follows:] 

Prepared Statement of Chairman Vernon J. Ehlers 

Good morning. Welcome to today’s hearing on the Environmental Protection Agen- 
cy’s fiscal year 2005 Science and Technology budget. 

The role of science and technology at EPA is more important than ever. EPA is 
being asked to address increasingly complex technical questions in its regulatory 
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processes and in its search for emerging environmental problems. That is why I am 
surprised and concerned about the Administration’s proposal to cut $92 million, 
which is nearly 12 percent, from EPA’s Science budget. It is hard to understand 
why, even in this tight budget time, the science budget would deserve such a sub- 
stantial cut. 

Today, the Subcommittee hopes to learn why the Administration proposes a cut 
of $35 million, or 35 percent, to EPA’s competitive, external research grants to col- 
leges, universities and other researchers, as well as the impact of those reductions. 
This proposed cut is in the Science to Achieve Results grant program, affectionately 
referred to as STAR grants. It would mean less extramural research on ecological 
systems, pollution prevention, endocrine disruptors and mercury, among other top- 
ics. 

Today, we are also interested in understanding the extent to which the Office of 
Management and Budget’s effort to assess the performance of government programs 
under its new Program Assessment Rating Tool, called PART, led to these proposed 
cuts. As a strong supporter linking performance to funding decisions, I look forward 
to a healthy discussion of how we can best evaluate environmental research and de- 
velopment programs. 

The PART is playing an increasingly important role in the Administration’s budg- 
et decisions. It has been applied to hundreds of federal programs in the last two 
years and will be applied to the remainder over the next four. With this in mind, 
it is critically important for this committee, and the Congress overall, to understand 
how this assessment tool is being used. 

I would also like to mention my concern about the $8.3 million reduction the Ad- 
ministration proposes that would eliminate EPA’s research on building decon- 
tamination. EPA has a unique role in cleaning contaminated buildings and it is not 
clear why this work is being cut, or whether another agency, such as the Depart- 
ment of Homeland Security, will pick it up. The Full Science Committee plans to 
address this question further at our joint hearing with the Select Committee on 
Homeland Security on March 25th. 

We have an excellent panel of witness today and I expect that we will learn a 
great deal. I’m looking forward to the testimony. 

Mr. Miller. Thank you, Mr. Chairman. 

I am pleased to be here this morning to open this hearing in Mr. 
Udall’s absence. I am also very pleased to hear how much common 
ground there is between my views and those just expressed by the 
Chair. 

I represent many of EPA’s scientific staff who work at the Re- 
search Triangle Park in North Carolina. My interest, however, is 
more than parochial. I think I have the same interest in EPA’s re- 
search and development funding that every American has or should 
have. In the 1960s, the Federal Government moved several air pol- 
lution-related environmental research facilities to the Research Tri- 
angle Park in North Carolina. This later evolved into a campus for 
the EPA, consisting of 400 laboratories. The laboratories focus on 
a wide range of research issues facing our nation, including air, 
health, exposure and environmental information. The Research Tri- 
angle is home to the EPA’s largest operation outside of Wash- 
ington, with a labor force of more than 2,500, and a $250 million 
annual contribution to North Carolina’s economy. 

You do not have to spend a lot of time on environmental issues 
to realize that they are very complicated and often very conten- 
tious. We have made great strides in balancing and recognizing the 
need to balance and in balancing economic development and envi- 
ronmental protection, and the ability to strike that balance is 
largely because of our investment in research and development. 
Protection of the environment is central to the protection of public 
health. Clean air and clean water are not luxuries. They are neces- 
sities. As a former member of the North Carolina legislature, I am 
very much aware of the problems caused by air pollution. Every 
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summer, air pollution causes 240,000 asthma attacks, 6,300 emer- 
gency room visits, and 1,900 hospital admissions in North Carolina. 
Almost one in five emergency room admissions in North Carolina 
is for pediatric asthma. That is a terrified parent taking their child 
to the emergency room because their child cannot catch their 
breath. 

I will continue to work to improve that environmental health 
problem to protect North Carolina’s children and North Carolina 
citizens against air pollution. I cannot reconcile the right to clean 
air and clean water and the constant call to base regulations and 
environmental policy on sound science with the Administration’s 
budget request for this year for research and development at EPA. 
The request essentially guts R&D funding. From her written testi- 
mony, I understand that Dr. Matanoski of EPA’s Science Advisory 
Board shares that opinion. 

The Administration has requested cuts in research and develop- 
ment funding for EPA to about 12 percent below the appropriated 
level for this year. Now last year’s request proposed a cut to EPA’s 
Science and Technology accounts by almost five percent. The Ad- 
ministration is obviously going in the wrong direction. 

The specific program cuts contained in this budget are unaccept- 
able and difficult to understand. EPA’s extramural grant program, 
which I believe the Chairman just mentioned, the Science to 
Achieve Results program, which has received high marks in exter- 
nal reviews by the Science Advisory Board and the National Acad- 
emy of Sciences, is rewarded with a disproportionate cut of about 
one-third of its budget. During the same month, the Senate was 
closed due to contamination from ricin; this Administration re- 
leased this budget terminating its research on building decon- 
tamination. 

This budget request does not serve the needs of my constituents, 
I think the constituents of many on the Committee or in the Con- 
gress, or maintain a healthy research and development program 
within the EPA. If this is the result of applying the Office of Man- 
agement and Budget’s new budget evaluation tool, the PART, clear- 
ly, the tool is deeply flawed. 

I do welcome our panel of experts, our witnesses, and I look for- 
ward to hearing your testimony today. Thank you. 

[The prepared statement of Mr. Miller follows:] 

Prepared Statement of Representative Brad Miller 

I am pleased to be here this morning to open this hearing. I have the privilege 
of representing many of EPA’s fine scientific staff that works at Research Triangle 
Park in North Carolina. As such, I have not only a general interest in EPA’s re- 
search and development funding, but a hometown one as well. 

In the 1960’s, the U.S. government moved several air pollution-related environ- 
mental research functions to Research Triangle Park, North Carolina. This later 
evolved into a campus for the EPA consisting of 400 laboratories. The laboratories 
focus on a wide range of research issues facing our nation including air, health, ex- 
posure and environmental information. The Research Triangle is home to the EPA’s 
largest operation outside of Washington with a labor force of more than 2,500, and 
a $250 million annual contribution to North Carolina’s economy. 

You don’t have to spend much time on environmental issues to recognize they are 
contentious and complex. We have made great strides in balancing economic devel- 
opment and environmental protection because we have invested in research and de- 
velopment. Protection of the environment is central to the protection of public 
health. Clean air and clean water are not luxuries. They are necessities. As a former 
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member of the North Carolina legislature, I am acutely aware of the problems 
caused by air pollution. Every summer, air pollution causes 240,000 asthma attacks, 
6,300 ER visits, and 1,900 hospital admissions in our state. I will continue to fight 
this environmental and health problem. 

I cannot reconcile the right to clean air and water and the constant call to base 
regulations and environmental policy on sound science with the Administration’s FY 
2005 request for research and development at EPA. The request guts R&D funding. 
I note from her written testimony, that Dr. Matanoski of EPA’s Science Advisory 
Board shares my opinion. 

The Administration’s request cuts research and development funding for EPA by 
about 12 percent below the appropriated level for this year. Last year’s request pro- 
posed a cut to EPA’s Science and Technology accounts by almost five percent. The 
Administration is going in the wrong direction. 

The specific program cuts contained in this budget are unacceptable and difficult 
to understand. EPA’s extramural grant program — the Science to Achieve Results 
program — which has received high marks in external reviews by the Science Advi- 
sory Board and the National Academy of sciences is rewarded with a dispropor- 
tionate cut of about one third. During the same month the Senate was closed due 
to contamination from ricin, the Administration released this budget terminating its 
research on building decontamination. 

This budget request does not serve the needs of our constituents or maintain a 
healthy research and development program with the EPA. If this is the result of 
applying the Office of Management and Budget’s new budget evaluation tool — the 
PART — clearly this tool is deeply flawed. 

I welcome our panel of witnesses and I look forward to hearing your testimony. 

Chairman Ehlers. If there is no objection, all additional opening 
statements submitted by the Subcommittee Members will be added 
to the record. Without objections, so ordered. 

I have to issue a correction. My staff said that I referred to the 
cut as $92 million in my opening statement. I referred to it as $22 
million. It is actually $92 million. I hope that will be my only sen- 
ior moment today, but I wanted to make clear that it is a very 
large amount of money. 

At this time, it is my pleasure to introduce our witnesses. We are 
honored today to have with us Mr. Clay Johnson, the Deputy Di- 
rector of Management at the Office of Management and Budget. 
We welcome you. It is good to see you in this role. I spent many 
times on the telephone with you in your first job in the personnel 
office, trying to recruit good scientists to work in the government 
and the Bush Administration, and it is a pleasure to see you in this 
role now, so thank you for coming. 

Next, we have Dr. Paul Gilman, who is no stranger in this room. 
He has appeared here frequently. He is the Assistant Adminis- 
trator for Research and Development at the United States Environ- 
mental Protection Agency, of course, always known by its acronym 
EPA. 

Next, Mr. Paul Posner. Is it Posner or Posner? Posner. Posner. 
Sorry. Mr. Paul Posner is the Managing Director for Budget and 
Intergovernmental Relations at the General Accounting Office, 
which we all know by GAO. He offers — oversaw GAO’s recent re- 
port on how PART is being implemented. 

Next, we are pleased to have Dr. Matanoski, who is a Professor 
in the Department of Epidemiology at Johns Hopkins University. 
She also chaired the EPA Science Advisory Board Review of EPA’s 
Fiscal Year 2005 Budget. While I introduce Dr. Matanoski, I would 
also like to recognize Dr. William Glaze, who is the current chair 
of EPA’s Science Advisory Board, who has helped lead the budget. 

And finally, Dr. Denson is a Professor in the Department of En- 
gineering at the University of Delaware. He is also a Member of 
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the National Academy of Science’s Panel that reviewed the EPA’s 
Science to Achieve Results Research Grants Program, better known 
by its acronym STAR. And now I would like to introduce not only 
the witnesses, but all the acronyms. We can soon proceed to the 
testimony. Our witnesses I presume have been informed that spo- 
ken testimony is limited to five minutes each, after which the 
Members of the Committee will have five minutes each to ask 
questions. If you cannot complete your statement, obviously, your 
written statement will go in the record. If you do not feel you have 
had a chance to say everything you wanted to say in your opening 
oral statement, you will have ample opportunity during the ques- 
tions to raise any points you believe should be raised. 

At this point, we will open our first round of questions. The 
Chair recognizes himself for five minutes. Oh, I am sorry. That is 
my second senior moment. It is a bad day today. I was ill last 
night, but that is no excuse. A Member of the Minority wants me 
to lose count. I suppose he would like me to forget who is the mi- 
nority here too. Okay. We will start with Mr. Johnson. 

STATEMENT OF MR. CLAY JOHNSON, III, DEPUTY DIRECTOR 

OF MANAGEMENT, OFFICE OF MANAGEMENT AND BUDGET 

Mr. Johnson. Mr. Chairman, Mr. Miller, thank you for having 
me. 

We, all of us, are interested and working very hard to make the 
Federal Government results oriented, and I would suggest, Mr. 
Miller, that in fact, we are hitting in the right direction, not the 
wrong direction. The right direction is let us pay attention to the 
results we are getting for the money we are spending, and not how 
much money we are spending. And I think this is your interest and 
our interest as well, as I think this is the direction that in fact the 
Federal Government is heading. 

EPA is one of the leaders in the Federal Government at focusing 
on results. They are particularly strong in the financial manage- 
ment area in the way we keep score, the score card. They are one 
of the few agencies that have achieved green, which just means 
they are using accurate and timely financial information with 
which to make regular, frequent management decisions. So they 
are one of the agencies we are very, very proud of in terms of the 
accomplishments they have made. 

We are here to talk about two things, as I understand it. One 
of them is the PART and one of them is the specific decision that 
has been with regard to ecological research and the Pollution Pre- 
vention Program. The PART helps agencies look at their programs 
with consistency. I would suggest that it is a valuable tool now. I 
believe this is GAO’s assessment of the program, and it is a tool 
that will get — it is a device that will get better every year. There 
is nothing magic about the PART. There is nothing sacrosanct 
about the 25 or 30 questions we ask. What is magic about it is it 
is a device now that will get better over time that — with which we 
can look at programs and ask ourselves key management structure 
and results, questions about programs in a consistent fashion, and 
focus more and more with increasing proficiency as to what we are 
getting for our money. Nothing happens automatically, as a result 
of a PART evaluation, but the information that comes out of the 
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PART assessment is used to help inform decisions about how to 
better manage, better structure, and better fund programs, includ- 
ing research programs. 

The two programs in particular that are the primary focus of this 
hearing, the Ecological Research and Pollution Prevention Pro- 
grams, were assessed with the use of the PART. They were consid- 
ered to be less results-oriented than they could be or than other 
programs of a similar nature. In fact, EPA’s Prevention, Pesticides 
and Toxics Program were considered to be most results-oriented, 
and in fact, the money that is recommended not be spent, the re- 
ductions that were recommended for these two programs in fact is 
being put in this other program, where the belief is that the return 
for the taxpayer on the $30 million will be greater than it would 
be if it was spent in the other programs. These two programs are 
well-funded at the recommended 2005 levels. They are reasonably 
well-funded programs. Therefore, we believe we do not significantly 
impair the focus of these research programs with these reductions, 
but we do produce a greater return for the taxpayer. 

The PART was used to inform this budget proposal, but there 
was nothing automatic that flowed out of this PART assessment. 
These programs were not as results-oriented as they could be, or 
we thought should be. The other obviously major part that went in 
this funding recommendation was the very tight fiscal situation we 
find ourselves in, trying to produce very, very tight budgets, not re- 
lated to Homeland Security and Defense. But I look forward to 
your questions later on. Thank you. 

[The prepared statement of Mr. Johnson follows:] 

Prepared Statement of Clay Johnson, III 

Introduction 

Thank you, Mr. Chairman, Members of the Committee, for inviting me to testify 
this morning. I want to discuss with you our assessment of the Environmental Pro- 
tection Agency’s (EPA) research programs and describe how the President’s Manage- 
ment Agenda helps federal agencies get greater results on behalf of the American 
people. 

We, all of us, are in the process of making the Federal Government results-ori- 
ented. We here in Washington tend to focus on the amount of money we’re spending 
as a validation for how much the Federal Government is committed to an objective. 
As a part of becoming results-oriented, however, we are now focusing more heavily 
on the results we achieve on behalf of the American people. With just a little help 
from OMB, agencies are asking whether they are achieving their objectives as effec- 
tively and efficiently as possible. EPA is a leader in this effort. 

EPA is as advanced as any agency in government in having and using accurate 
financial information to make day-to-day decisions about program management. For 
example, EPA negotiates performance commitments with its grantees and provides 
resources based on those commitments. EPA regularly monitors grantees’ perform- 
ance and expenditures and, if a grantee isn’t meeting its commitments, EPA may 
withhold resources from the non-performers and redirect those resources to grantees 
that are meeting their commitments. 

The Program Assessment Rating Tool 

Applying the Program Assessment Rating Tool (PART) is one of the ways we are 
becoming results-oriented. The PART is a series of questions that assesses the pur- 
pose, strategic planning, management, and performance of individual programs. 
Programs must demonstrate that their purpose is clear, that they set aggressive, 
outcome-oriented long- and short-term goals, that they are well managed, and that 
they achieve results. With this tool, we are assessing the performance of every fed- 
eral program, and if it is not working as intended, we are trying to do something 
about it. 
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The Administration has used the PART to assess 400 programs so far, rep- 
resenting approximately $1 trillion in federal spending. We are using these assess- 
ments not only to guide our budget decisions, but also to improve the performance 
and management of the government’s programs. The purpose of asking whether pro- 
grams are working is to figure out how to fix them, not whether to spend more or 
less on them. 

Ecological Research and Pollution Prevention PARTs 

As you know, OMB and EPA assessed EPA’s Ecological Research program and 
Pollution Prevention and New Technologies program using the PART. According to 
the assessment, the Ecological Research program: 

• did not adequately coordinate the expenditure of resources with other EPA 
offices or other agencies; 

• lacks adequate annual measures of its performance; and 

• does not have sufficient evaluations of its performance. 

Like nearly 40 percent of the programs evaluated using the PART, the principal 
finding for the program was the lack of adequate performance measures. Therefore, 
EPA has committed to finding the right measures for this important program. The 
President has requested $110 million for this program in his FY 2005 Budget, down 
from $132 million in FY 2004. 

According to the assessment, the Pollution Prevention and New Technologies pro- 
gram: 

• has not addressed findings made by independent evaluations; and 

• has not developed adequate measures of its performance. 

As a result of these findings, EPA has committed to developing adequate perform- 
ance measures and addressing findings made in previous independent evaluations. 
The President has requested $36 million for this program in his FY 2005 Budget, 
down from $42 million in FY 2004. 

Why reduce funding for these programs? 

Both the Ecological Research and Pollution Prevention programs were “unable to 
demonstrate results,” which clearly influenced funding decisions related to the pro- 
grams. Especially in a year like this one, when resources are constrained, we should 
be directing resources to those programs that can achieve the most for the money. 
EPA and OMB used the PARTs for the Ecological Research and Pollution Preven- 
tion programs as one factor in making budget decisions about those programs and 
to focus resources on the programs most effective in helping EPA accomplish its 
mission. 

As I’ve mentioned, the Pollution Prevention research program could not show 
whether the tools it is developing are used by industry, and, if so, to what extent 
they are used. Also, previous independent evaluations of the Pollution Prevention 
research program concurred with the PART review, especially in the areas of stra- 
tegic planning and measurable results. On the other hand, a similar program in the 
EPA’s Office of Prevention, Pesticides, and Toxics was able to show that industry 
reduced its use and emissions of toxic chemicals through the use of tools and meth- 
ods developed by the program. We consider reductions in pollution to be one of the 
highest-level outcomes of an environmental program’s performance. Therefore, we 
redirected funds to the pollution prevention program so EPA can continue to achieve 
pollution reduction, thereby positively impacting the quality of public health and the 
environment. Despite redirection of a small amount of funds from pollution preven- 
tion research to OPPTS’s program, the Administration maintained a large amount 
of funding for the pollution prevention research program to assist it, among other 
things, in developing performance measures. 

This is our rationale for funding decisions related to EPA’s research programs. I 
will leave to Dr. Gilman a more robust discussion of how these funding decisions 
were applied to specific components of the research programs. 

Research and Development and the Investment Criteria 

The Government’s investment in research & development, not only in the environ- 
mental arena but elsewhere, is substantial. But in a time of constrained resources, 
it is imperative that we invest in R&D wisely. In recognition of the special chal- 
lenges that measuring R&D programs present, and leveraging work done by the Na- 
tional Academies of Science, the Administration developed its R&D Investment Cri- 
teria, which were incorporated into the PART. These criteria are some of the things 
we look at when assessing the value of particular R&D programs: 



38 


• Relevance. Programs must be able to articulate why they are important, rel- 
evant, and appropriate for federal investment; 

• Quality. Programs must justify how funds will be allocated to ensure quality; 
and 

• Performance. Programs must be able to monitor and document how well the 
investments are performing. 

As noted in our PART evaluations, the programs we assessed could improve the 
ways they measure their performance. The three EPA programs we assessed cover 
important issues, and receive funding totaling approximately $210 million. We 
strongly believe that programs with federal funding of this magnitude should be 
able to monitor and document how these investments are performing. There are 
other equally important programs that are receiving similar levels of funding, but 
whose results are more measurable. For example, the Department of Energy’s Wind 
Energy program, with proposed FY 2005 funding of $42 million, can demonstrate 
its contributions to the commercial success of wind energy use throughout the 
United States. The Federal Aviation Administration’s Research, Engineering and 
Development program, with proposed FY 2005 funding of $117 million, has set a 
long-term goal to produce turbulence forecasting products that allow pilots to avoid 
hazardous flight conditions while improving safety and ensuring efficient airspace 
use. 

The Future of the PART 

The PART is a vehicle for improving program performance. It is just a tool to 
achieve the goals laid out by Congress in the Government Performance and Results 
Act (GPRA). GAO has made a number of recommendations for improving the PART, 
the vast majority of which we agree with and are addressing. For example: 

• With respect to centrally monitoring PART recommendations, we have pro- 
vided a simple format for agencies to follow when reporting the status of rec- 
ommendation implementation to OMB and I receive these reports semi-annu- 
ally. We will continue to refine this process so that sufficient attention is 
given to recommendation follow-up. 

• As the PART relies on separate evaluations of evidence of a program’s suc- 
cess, we agree with GAO that the judgment about what constitutes a suffi- 
cient evaluation should be based on the quality, in addition to the independ- 
ence, of the evaluation. 

• One of the greatest opportunities for the PART is to compare the performance 
of, and share best practices among, like programs across government. We will 
continue to use the PART for that purpose. 

• We will continue to improve agency and Executive Branch implementation of 
GPRA by insisting GPRA plans and reports meet the requirements of this im- 
portant law and the high standards set by the PART. 

• We are clarifying the PART guidance so that it is well understood by those 
who have to use it, as well those who have to administer it. We will continue 
to assess completed PARTs to ensure they are completed consistently by 
agencies and OMB. 

Conclusion 

The PART is a valuable tool now, as the General Accounting Office and others 
have asserted, and it will get better each year. As more and more program assess- 
ments are conducted, the vast majority of budget and management decisions will 
be significantly influenced by information about how programs are performing. 
Agencies, including EPA, will be better able to describe to Congress and the tax- 
payer what his or her funding is purchasing and will be managing so that each year 
improvements in efficiency and service delivery can be documented. 

Biography for Clay Johnson, III 

Clay Johnson is the Deputy Director for Management at the Office of Manage- 
ment and Budget. The Deputy Director for Management provides government-wide 
leadership to Executive Branch agencies to improve agency and program perform- 
ance. Prior to this he was the Assistant to the President for Presidential Personnel, 
responsible for the organization that identifies and recruits approximately 4,000 
senior officials, middle management personnel and part-time board and commission 
members. 
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From 1995 to 2000, Mr. Johnson worked with Governor George W. Bush in Aus- 
tin, first as his Appointments Director, then as his Chief of Staff, and then as the 
Executive Director of the Bush-Cheney Transition. 

Mr. Johnson has been the Chief Operating Officer for the Dallas Museum of Art 
and the President of the Horchow and Neiman Marcus Mail Order companies. He 
also has worked for Citicorp, Wilson Sporting Goods and Frito Lay. 

He received his undergraduate degree from Yale University and a Master’s degree 
from MIT’s Sloan School of Management. In Austin, he helped create the Texas 
State History Museum, and was also an Adjunct Professor at the University of 
Texas Graduate School of Business. In Dallas, he served as President of the Board 
of Trustees for St. Marks School of Texas, and as a Board Member of Equitable 
Bankshares, Goodwill Industries of Dallas, and the Dallas Chapter of the Young 
Presidents Organization. 

Chairman Ehlers. Thank you. 

Dr. Gilman. 

STATEMENT OF DR. PAUL GILMAN, ASSISTANT ADMINIS- 
TRATOR FOR RESEARCH AND DEVELOPMENT, UNITED 

STATES ENVIRONMENTAL PROTECTION AGENCY 

Dr. Gilman. Thank you, Mr. Chairman, Members of the Com- 
mittee. Thanks for the opportunity to be here. I am here as the As- 
sistant Administrator for the Office of Research and Development. 
I also serve as Science Advisor for the Agency. 

The Office of Research and Development is a service element to 
the various programs of the Agency. It does both basic and applied 
research. Roughly half basic, half applied. It does research in 
human health, public health and environmental and ecological 
sciences. Again, approximately a 60/40 split, in terms of human 
health and environmental health. There is a great emphasis on 
quality in the research program. It ranges from the very earliest 
stages of planning. We have instituted a planning process that de- 
velops multi-year plans for each of the major research areas. Those 
plans are developed in collaboration with our customers, if you will, 
the various programs of the agency; water, air, land, and coordi- 
nated with other federal agencies and outside entities, as well. 

We have gone from an Agency that was viewed as a laggard in 
the use of peer-review to, I think today, an agency viewed as a 
leader in the use of peer-review, and the Extramural Grants Pro- 
gram, as you will hear later from the witness representing the Na- 
tional Research Council, is a well-respected program as well. We 
place a great emphasis on collaborative research with other federal 
agencies in order to stretch our dollars and to avail ourselves of 
their expertise. Probably a very good example of that collaborative 
effort is a report we just released two days ago on the health of 
our nation’s coastal waters, where we collaborated with 28 different 
states, a number of federal agencies; NOAA, Department of Inte- 
rior, Agriculture, to really produce the first scientifically support- 
able assessment of the quality of coastal waters that we have had 
in a comprehensive way. 

The programs of particular matter, ecological research and pollu- 
tion prevention all have tough goals set for them by our Agency, 
and embedded in our multi-year plans. Like providing the data and 
tools to predict, measure, reduce and meet the standards for partic- 
ulate matter, as well as producing that report I just mentioned on 
the coastal health of our ecosystems. Yet, we are still challenged 
to provide measures that truly get into the performance of those 
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programs. I am committed to working with the Office of Manage- 
ment and Budget and others to create long-term annual and effi- 
ciency measures that capture the important work our programs are 
doing. In the end, these measures will help advance our programs 
by demonstrating the value of our achievements very clearly. 

Let me say something about some recent accomplishments to 
demonstrate the breadth of the work that we are doing. These 
goals and accomplishments all draw on the STAR program that is 
the subject of this hearing. We have recently done work that is di- 
rectly applicable to protecting water quality, a mission for the 
Agency, that also has Homeland Security implications towards bet- 
ter understanding of how water distribution systems actually work, 
not the pre-treatment but the post-treatment side of the water sys- 
tem. We are working to improve our air models, to improve their 
performance and their accuracy. We are working to develop DNA- 
based technologies for the identifications of things like common 
household molds that are deleterious to our health. And a develop- 
ment like that is the kind of thing we are very proud of because 
we are in the process of licensing it to a number of companies, with 
over 15 companies licensing that particular technology. 

Again, Mr. Chairman, it is a challenge for us to relate the results 
of our research on inherently long-term research to measures for 
environment and public health outcomes. We are committed to 
doing it. We have been trying to develop some approaches in col- 
laboration with our Inspector General. We are also very interested 
in continuing our discussions with the Office of Management and 
Budget to utilize some different approaches to the evaluation of 
these programs, and I think they will prove to be effective. 

Thank you, Mr. Chairman. 

[The prepared statement of Dr. Gilman follows:] 

Prepared Statement of Paul Gilman 


Introduction 

Good morning Mr. Chairman and Members of the Subcommittee. I am honored 
to appear before you today to discuss the Fiscal Year (FY) 2005 budget request for 
the U.S. Environmental Protection Agency’s (EPA) Office of Research and Develop- 
ment (ORD), and to share with you the uniqueness and success of ORD’s research 
program from my perspective as both the Assistant Administrator for ORD and the 
EPA Science Advisor. 

The President’s FY 2005 budget request for ORD is $572.2 million. This includes 
funding for ORD’s in-house program carried out by 1,975 employees, who account 
for 11 percent of EPA’s workforce. In addition, the budget request supports our 
Science to Achieve Results (STAR) research grants program. Together, our in-house 
and STAR programs allow our nation’s brightest scientists to apply their talents and 
knowledge to solve environmental science problems. My testimony highlights the 
contributions we and our partners have made and describes changes to the Agency’s 
research budget for STAR research in FY 2005. 

ORD’s Unique Contributions 

ORD conducts leading-edge research and fosters the use of science and technology 
in environmental decisions in support of EPA’s mission to protect human health and 
safeguard the environment. This research tackles problems to which solutions will 
have both immediate and long-term public health and environmental benefits. The 
advancement of science and the development of answers to questions posed by envi- 
ronmental issues makes ORD unique among federal research agencies. No other fed- 
eral agency has a comprehensive research program devoted to improving our under- 
standing of both public health and environmental impacts. No other agency is re- 
searching these issues in an integrated fashion. In addition, no other agency can 
claim as large an impact on ensuring EPA’s decisions are informed by the strongest 
possible science. To further strengthen our science program, EPA has been imple- 
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meriting the National Research Council (NRC) recommendations in its 2000 report, 
“Strengthening Science at the U.S. Environmental Protection Agency: Research 
Management and Peer Review Practices,” as I will describe below. In sum, ORD is 
conducting leading-edge research that informs the risk-based environmental deci- 
sion-making of EPA’s program and regional offices and helps States and Tribes de- 
cide how best to implement these policies. 

Ensuring these decisions are based on sound science requires relevant, high qual- 
ity, integrated, leading-edge research in human health, ecology, pollution prevention 
and control, and socio-economics. To maintain both short- and long-term relevance 
to EPA’s mission, ORD’s scientific research activities are mainly focused on applied 
research, which is problem-driven and, to a lesser extent, basic research. To ensure 
the quality of our research program, ORD uses a coordinated, cooperative research 
planning process; rigorous, independent peer review; and interagency partnerships 
and extramural grants to academia that complement EPA’s own in-house scientific 
expertise. We have a uniquely integrated research program in that we address both 
human and ecological endpoints, conduct research across the risk assessment/risk 
management paradigm, have expertise across scientific disciplines and within the 
different environmental media, and draw from expertise in other agencies, organiza- 
tions, and academia. Lastly, ORD keeps a leading edge in research by focusing our 
efforts and resources on those areas where EPA can add the most value toward re- 
ducing uncertainty in risk assessments and enhancing environmental management. 

The following are a few examples of our more recent accomplishments. ORD re- 
searchers: 

• Collaborated with the Department of Homeland Security, Department of En- 
ergy, Department of Defense, and Centers for Disease Control to strengthen 
water security, develop rapid risk assessment techniques, and develop Build- 
ing decontamination methods. 

• Partnered with 24 marine coastal States, four territories, and other federal 
agencies through the Environmental Monitoring and Assessment Program’s 
National Coastal Assessment, to conduct sampling of estuaries using prob- 
abilistic methods. 

• Collaborated with EPA’s Office of Environmental Information to deliver the 
draft Report on the Environment, the first-ever national picture of U.S. envi- 
ronmental quality and human health using science-based indicators. 

• Developed the Computational Toxicology Program, which has moved EPA to 
the leading edge in the use of genetics, genomics, and computation for envi- 
ronmental protection. 

• Completed an evaluation of Superfund clean-up technologies citing 143 suc- 
cessfully demonstrated technologies and $2.6 billion in total inflation-adjusted 
cost savings. 

• Continued our tradition of leadership in the use of external scientific exper- 
tise to enhance the quality and relevance of our scientific products, by relying 
on the processes of peer participation and peer review. 

I am proud of these accomplishments and the others I will identify later. They 
are the direct result of careful research planning that relies on the active involve- 
ment of the Agency’s program and regional offices, as well as outside peer input. 

Research Planning 

The President’s budget request for FY 2005 will allow us to build upon these ac- 
complishments by continuing a research program that directly serves EPA’s mission. 
EPA’s science and technology efforts are aligned with the Agency’s strategic goals, 
and we now have gone a step further by including science objectives within each 
of EPA’s five strategic goals. ORD created these science objectives in collaboration 
with EPA’s program and regional offices, to ensure that we produce the right sci- 
entific and technical information to meet EPA’s programmatic needs and thereby 
advance the Agency’s mission. 

The alignment of our science and technology program with EPA’s strategic goals 
is carried forward into ORD’s planning of our research and development program. 
We have divided our R&D program into topical areas, each of which is guided by 
a multi-year research plan (the plans can be found at www.epa.gov / osp / myp.htm). 
Each multi-year plan contains long-term research and development goals for the 
next 5-10 years that tie back to EPA’s strategic goals, and are supported by annual 
performance goals and measures. Every multi-year plan, and the goals and meas- 
ures that comprise the plan, is developed in concert with colleagues across EPA and 
in consultation with our stakeholders and the broader scientific community. The 
plans also undergo expert, external peer review by EPA’s Science Advisory Board 
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(SAB) and ORD’s Board of Scientific Counselors (BOSC). Both groups have endorsed 
this research planning process. 

The multi-year plans are “road maps” that mark the progress our research pro- 
grams have already made, as well as lay out the new directions we are taking to 
adjust as changes occur in the complex scientific landscape ahead. Developing this 
road map requires identifying a logical progression of scientific research to be con- 
tributed by EPA and its partners. This progression is defined in each multi-year 
plan using “logic models” that demonstrate how research results contribute to EPA’s 
desired long-term outcomes of improved human and ecosystem health. By following 
the logic diagram, one can begin to see how each research project contributes to the 
achievement of the long-term outcome. For illustration purposes, I have attached 
the logic diagram from our Particulate Matter (PM) multi-year plan. I discuss logic 
models in greater detail later in this testimony. 

The multi-year research plans help EPA maintain its focus on high-priority 
science issues. They also assist in coordinating research efforts across the environ- 
mental science community, including other federal entities; State, tribal, and local 
governments; international organizations; and academia. Such coordination is essen- 
tial. EPA’s science and technology budget is only a small fraction of the total annual 
expenditures on environmental research, so leveraging our efforts with others — and, 
most important, identifying the appropriate niche for EPA’s science and technology 
programs — is necessary for our doing the right science in a fiscally responsible man- 
ner. 

Independent scientific bodies have lauded EPA’s process for planning its research 
efforts. In its 2000 publication, Strengthening Science at the U.S. Environmental 
Protection Agency, the National Research Council stated, “Our committee expects 
that ORD’s recent efforts in multi-year planning will contribute greatly to research 
program continuity and the achievement of strategic goals, and ORD merits com- 
mendation for these initiatives.” Four years later, I can state with confidence that 
our research planning process is meeting — and perhaps exceeding — the NRC’s ex- 
pectations. 

I wish to discuss two of our research programs — airborne particulate matter and 
ecosystem protection — to illustrate how EPA’s science complements the scientific 
work of others, to advance scientific understanding and inform the decisions that 
solve environmental problems. Both of these research programs were evaluated 
using the Program Assessment Rating Tool (PART). The principles and practices ap- 
plied in the particulate matter and ecosystem protection programs are those used 
in each of EPA’s research and development programs. 

Particulate Matter 

Among the most serious environmental problems affecting the health of Ameri- 
cans is exposure to airborne particulate matter. Based on the best science available 
to us, these exposures contribute to the premature deaths of tens of thousands of 
Americans annually, as well as the hospitalization of children and adults for dis- 
eases such as asthma. This has been documented in the Office of Management and 
Budget’s (OMB) “Thompson Report” (68 Fed. Reg. 5492, 5499 (2003)). To protect the 
public against these effects, the Clean Air Act calls for the promulgation and peri- 
odic review of National Ambient Air Quality Standards, or NAAQS. In the late 
1990s, after such a review yielded new standards for fine PM (particles less than 
2.5 microns in diameter), Congress authorized and appropriated funds to EPA for 
a greatly expanded PM research program, to be guided by advice from the National 
Research Council. I would like to describe how we have organized this program and 
share what we have learned. 

To deliver the best science needed to inform sound public policy decisions, we 
have worked with our Agency partners in the Office of Air and Radiation and the 
regions to develop a multi-year plan for PM research that looks forward a little 
more than a decade. This plan, which will be peer reviewed by the EPA Science Ad- 
visory Board later this year, describes research activities in two major areas: (1) PM 
health effects and exposure, to guide future reviews of the NAAQS to refine the type 
and amount of PM that needs to be controlled to protect public health; and (2) im- 
plementation tools, so that EPA, the States and the tribes, and the private sector 
can ensure that these standards are met. 

The PM multi-year plan integrates the strengths of our in-house scientists with 
those of the external scientific community, through the extensive use of our STAR 
research grants program, including the support of five PM Research Centers. In ad- 
dition, EPA’s researchers are coordinating their efforts with others in the public and 
private sectors, both domestically and internationally. For example, health research 
is being conducted overseas by several organizations, while in the United States, 
studies are being supported by industrial organizations including the Electric Power 
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Research Institute and the Coordinating Research Council through their support of 
the Health Effects Institute (co-funded by EPA). Recently, EPA, the National Insti- 
tute of Environmental Health Sciences, and the National Heart, Lung, and Blood 
Institute co-sponsored a workshop on the cardiovascular effects of environmental 
pollutants, and planning is now underway to develop joint Requests For Applica- 
tions in the area of cardiovascular effects of PM exposure. Through these and other 
mechanisms, EPA contributes to and keeps abreast of the scientific advancements 
and initiatives in the PM area. 

What have we learned since the setting of the 1997 NAAQS? Here are a few ex- 
amples: 

• In 1997, questions were raised about the legitimacy of findings showing asso- 
ciations between centrally-monitored PM and health effects. We now under- 
stand that these monitors actually do a good job at estimating population ex- 
posures, which has lent further credence to the health associations found in 
epidemiologic studies. 

• While we knew of these associations between PM and increased mortality in 
1997, we were at something of a loss to explain them biologically. Due to 
work done by both ORD in-house and STAR-supported extramural scientists, 
we now have several plausible hypotheses for the biological mechanisms lead- 
ing to those associations, including recent findings showing an effect of PM 
directly on the heart. 

• In 1997, we had a poor understanding of the chemical composition and size 
distribution of PM that correlated with health effects. Today, we have de- 
tailed profiles of the PM associated with many significant sources and geo- 
graphic areas, and we continue to refine our understanding about the specific 
types of sources responsible for these public health risks. 

While EPA’s PM research program has been a success, there continues to be more 
to learn, as described in the PM multi-year plan. One focus of the program in the 
coming years will be to integrate the methods of diverse disciplines to determine the 
specific types of PM, and their sources, that have the greatest effect on public 
health. This will allow future standards and control strategies to focus attention 
only on those sources of pollution that need to be addressed. Another major focus 
will be on understanding the effects of long-term exposures to PM, through the 
funding of a long-term epidemiologic study to be conducted as part of our STAR re- 
search program. Lastly, EPA will evaluate new technologies for reducing air pollu- 
tion, examining the ability of controls to reduce emissions of many pollutants at 
once. The results of these efforts will inform EPA’s future PM policies, to ensure 
these policies protect human health in the most effective ways. 

Ecological Research 

Current ecological management approaches have made important contributions to 
improved environmental quality through greatly reducing emissions of pollutants 
from point sources and waste disposal sites, and reducing the mishandling of toxic 
or hazardous chemicals and pesticides. Future ecological problems, however, will 
likely be more subtle, potentially more far-reaching, and require very different solu- 
tions. Examples include non-point source pollution control, regional-scale effects of 
air pollutants on aquatic ecosystems, dislocations in ecologically and economically 
important species due to invasions by non-indigenous species, and the cumulative 
effects and synergistic interactions of multiple stressors on the health of aquatic 
species and communities. 

To deliver sound science for informed decision-making, EPA has focused its eco- 
logical research program to assess and compare risks to ecosystems, to protect and 
restore them, and to demonstrate progress in terms of ecological outcomes. The eco- 
logical research program also reflects the growing ethic of environmental steward- 
ship and the recognition that the implementation of these ecological management 
approaches will be largely community and sector-based, place-based, and perform- 
ance-based. 

Environment and natural resource research is coordinated government-wide 
through the Committee on Environmental and Natural Resources (CENR). EPA is 
an active member on this committee, whose goal is to increase the overall effective- 
ness and productivity of federal research and development in environmental issues. 
Given the current fiscal constraints, EPA believes it is more important than ever 
for federal agencies to collaborate and coordinate research activities. EPA has a long 
history of collaborating ecosystem research with the National Science Foundation. 
EPA plans to continue, and wherever appropriate enhance, its coordination with 
NSF and other agencies. 
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ORD’s Ecological Research Strategy underwent interagency peer review by the 
CENR in June 1997, and external review by the Science Advisory Board’s Ecological 
Processes and Effects Committee in July 1997. The final Strategy, published in June 
1998, formed the basis for ORD’s Ecological Research Multi-Year Plan, which de- 
scribes how the Agency plans to align its resources to achieve the plan’s goals, in- 
cluding the integration of ORD’s in-house research efforts with those conducted by 
our STAR research grants program. 

The Ecological Research Multi-Year Plan lays out four critical scientific questions 
to be addressed and their associated research emphases and programmatic goals. 
These questions are: 

• What is the current condition of ecosystems and what are the trends in their 
condition over time? (Assessing condition) 

• How do natural ecological disturbances and human activities affect eco- 
systems? How can we most accurately diagnose the causes of ecosystem dete- 
rioration? (Diagnosis) 

• How can we reliably predict the vulnerability of ecosystems to harm from cur- 
rent resource development and management practices? How can we predict 
the most probable responses of ecosystems to best management and sustain- 
able development practices? (Forecasting) 

• How can we most effectively control risks and manage to protect ecosystems 
once they have been degraded? (Restoration) 

The PART evaluation on the ecological research program found that the program 
addresses a clear and continuing need and that it is generally well-managed, with 
adequate grantee and resource oversight. Its work has lead to accomplishments such 
as the Environmental Monitoring and Assessment Program (EMAP) National Coast- 
al Assessment accomplishments that I mentioned earlier. Additional examples in- 
clude: 

• Research methods and findings from ORD’s EMAP have enabled State and 
tribal water monitoring programs to obtain more reliable data on the ecologi- 
cal condition of their streams and rivers, at significantly lower cost than the 
methods they had been using. 

• ORD produced national guidelines on assessing ecological risks. For the first 
time, these guidelines extend the principles of EPA’s risk assessment para- 
digms to assessing and comparing risks to ecosystems. 

• STAR researchers have developed and applied integrated methods to model 
and evaluate the effect of stressors on water quality. These include develop- 
ment of models to: (1) estimate annual nutrient loading to Lake Tahoe from 
atmospheric deposition, precipitation, stream discharge, overland runoff, 
groundwater and shoreline erosion; (2) estimate how “build-out” in urbanizing 
watersheds affects nutrient cycling, water quality, and the ecological health 
of rivers and streams in Gwynns Falls, Maryland; (3) evaluate the effects of 
agricultural best management practices on stream flow, sediment, and nitrate 
loadings in the lower Minnesota River; and (4) contaminant loading and bio- 
accumulation in Lake Erie. 

As described in the Ecological Research multi-year plan, however, we are com- 
mitted to building upon these achievements, and in the future, the ecological re- 
search program will focus heavily on diagnosis, forecasting, and restoration re- 
search. This research will enable the Agency to implement performance oriented, 
place-based protection of ecological systems. Our challenge now is to translate these 
successes into performance measures that demonstrate the utility of the tools and 
other protocols that we develop. In particular, long-term goals are difficult for any 
environmental program to develop, even more so for an environmental research pro- 
gram. I am committed to working with OMB and others to create long-term, annual, 
and efficiency measures that capture the important work our program is doing. In 
the end, these measures will help advance our program by demonstrating the value 
of our achievements. 

Science Quality Across EPA 

While our comprehensive and collaborative research planning process guides EPA 
to do the right science, as EPA’s Science Advisor, I believe EPA’s integrated ap- 
proach to scientific quality makes sure that we also do the science right, not only 
in ORD but across the Agency. The three pillars of this approach are our Quality 
System, Information Quality Guidelines, and Peer Review Policy. 

EPA’s Quality System is the means by which we manage our scientific informa- 
tion in a systematic, organized manner. It provides a framework for planning, im- 
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plementing, and assessing the scientific work performed by EPA and for carrying 
out quality assurance and quality control activities. Each EPA organization develops 
a quality management plan that describes its quality system in terms of the organi- 
zational structure, policy and procedures, functional responsibilities of management 
and staff, lines of authority, and necessary interfaces for the planning, imple- 
menting, documenting, and assessing of all activities conducted. At the individual 
project level, we develop quality assurance project plans that describe the necessary 
quality assurance, quality control, and other technical activities that must be imple- 
mented to ensure that work outputs will satisfy the stated performance criteria. The 
goals of the EPA Quality System are to ensure that environmental programs and 
decisions are supported by data of the type and quality needed and expected for 
their intended use, and that decisions involving environmental technology are sup- 
ported by appropriate quality-assured engineering standards and practices. 

EPA recognizes that the Office of Management and Budget’s Information Quality 
Guidelines, together with our own Information Quality Guidelines issued in October 
2002, are an important step forward in the quest for quality. The OMB guidelines 
call for all federal agencies to develop quality performance goals, including proce- 
dures to assure quality before information is disseminated. In response to these 
guidelines, EPA has established a system for addressing complaints about the qual- 
ity of information that the Agency has disseminated. We now have more than a 
year’s worth of experience in addressing challenges to EPA information under the 
guidelines, and this experience has validated our belief that ensuring the quality of 
our scientific information is paramount to maintaining the integrity of, and the 
public’s confidence in, EPA’s policies and decisions. 

Consistent Agency-wide application of independent, expert scientific peer review 
has been an EPA priority for many years. Since issuing our peer review policy in 
1993, we have taken several major steps to support and strengthen the policy. But 
proof of a policy’s value lies in its implementation, and here also EPA has been very 
active to ensure that our peer review policy is not only understood across the Agen- 
cy, but is applied rigorously across EPA’s program and regional offices. EPA has in 
place a strong and extensive program for peer reviewing our scientific and technical 
work products. 

EPA’s approach to peer review is articulated in our policy, Peer Review and Peer 
Involvement at the U.S. Environmental Protection Agency. In addition to the policy, 
EPA has published a handbook that provides detailed guidance for implementing 
the policy. The Peer Review Handbook can be found at www.epa.gov / osp / spc 7 
2peerrev.htm. We believe this is one of the most advanced treatments of peer review 
for intramural research and scientific/technical analysis of any federal agency. 

Most of EPA’s scientific and technical work products now undergo peer review. In 
1995, the Agency identified 120 work products for peer review. In 2002, of 859 work 
products generated by or for EPA, only 111 were deemed, usually because of their 
repetitive or routine nature, not to be candidates for peer review. So, we see that 
nearly 90 percent of our scientific and technical work products receive internal or 
external peer review. And 90 percent of those peer-reviewed products received inde- 
pendent, external review. 

We were confident enough in the strength of our peer review program that we 
made it a cornerstone of our Information Quality Guidelines. Since issuing our pol- 
icy ten years ago, peer review has become a part of EPA’s culture, and its use is 
widespread across the Agency. Our challenge for the future is to continue the sig- 
nificant progress we have achieved to date and, not being content with the status 
quo, to look for ways to enhance the use of peer review as a tool for ensuring that 
EPA’s decisions are supported by a firm foundation of scientific and technical infor- 
mation. 

Doing the right science through forward-looking collaborative research planning, 
and doing the science right by adherence to information quality and peer review 
standards, have given EPA policy-makers relevant, timely, and credible scientific in- 
formation to guide Agency decisions. 

ORD — Making a Difference 

ORD scientists are committed to generating products of the highest quality to en- 
sure sound science informs Agency decision-making. Our successes have been nu- 
merous, and we continue to build upon them. I have highlighted below a sampling 
of such successes, to illustrate the depth, breadth, and relevance of our research pro- 
grams’ contributions to environmental science generally and to EPA’s mission in 
particular. As these examples demonstrate, ORD’s research program — as a major 
part of the entire EPA scientific endeavor — plays a critical role in protecting human 
health and safeguarding the environment. 
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• In July 2003, EPA conducted an important drinking water distribution field 
study to map the movement of contaminants in a water system. This research 
is helping water system managers and emergency responders better predict 
how a biological or chemical contaminant would react in a drinking water sys- 
tem. This study ties directly into EPA’s community support and homeland se- 
curity efforts. 

• ORD, working with academia, developed the first air quality model (Models- 
3/CMAQ) to use a “one atmosphere” approach to simulate the interactions 
among many air pollutants, which is necessary to achieve truly cost-effective 
air pollution control strategies. This work is critical for local air pollution 
forecasting, as well as supporting the Agency’s multi-pollutant control strate- 
gies. 

• Working with the Department of Energy and the National Oceanic and At- 
mospheric Administration, ORD is researching exposures to air pollutants in 
complex terrains, such as urban canyons created by high-rise buildings and 
complex traffic patterns. This research combines field monitoring with wind 
tunnel studies to refine exposure models that can be applied to different U.S. 
cities. 

• ORD developed toxicity methods for determining acute and chronic toxicity to 
plants, invertebrates, and vertebrates, using several different end points. 
ORD also participated in the development of the Whole Effluent Toxicity Test 
Methods Rule, which allows these methods to be used as a basis for decision- 
making in the National Pollutant Discharge Elimination Systems Program. 

• ORD developed analytical methods for Cryptosporidium and evaluated tech- 
nologies that could be used for removing Cryptosporidium from drinking 
water sources. ORD worked with the Office of Water to use these results in 
promulgating the Long-Term 2 Enhanced Surface Water Treatment Rule. 
This rule will protect drinking water consumers, including sensitive sub-popu- 
lations such as children, by avoiding Cryptosporidium incidents that have re- 
sulted in health impacts and even death in the past. 

• EPA’s cancer risk assessment prompted industry decisions to phase-out the 
use of chromated copper arsenate (CCA) treated wood in home settings, due 
to concerns of exposure to kids from decks and play equipment. ORD is work- 
ing with other EPA scientists to analyze exposures to homeowners and chil- 
dren from CCA-treated decks and play equipment and to evaluate coatings 
and sealants that can be used to reduce risk from exposure to CCA-treated 
wood. 

• ORD developed a DNA-based system that allows rapid identification and 
quantification of molds in a matter of hours, as opposed to current methods 
that require days or even weeks. The new technology can be used to detect 
the mold Stachybotrys, commonly known as “black mold,” and more than 50 
other possibly harmful molds. The new method has been licensed to 13 com- 
panies for use in detecting mold, and four additional licenses are pending. 

• EPA chairs the coordination of endocrine disruptor research across federal 
agencies through an interagency working group under the Committee on En- 
vironment and Natural Resources, under the President’s National Science and 
Technology Council. Through this interagency working group, EPA and its 
partners issued two joint solicitations for research proposals to address the 
critical data gaps of understanding the impact of endocrine disruptors on hu- 
mans and wildlife. 

Linking Research Results to Outcomes 

EPA recognizes that research findings — no matter how insightful or cutting- 
edge — cannot of their own accord achieve environmental outcomes. Achieving envi- 
ronmental outcomes depends on decisions made and actions taken by the Agency’s 
program and regional offices, as well as by our State and tribal partners. We are 
working with our Office of Inspector General (OIG) to develop better ways to de- 
scribe the link between our research program and environmental and public health 
outcomes. Our efforts are focused on the use of a logic model that was developed 
by the OIG. 

The OIG, in collaboration with the ORD, piloted using the logic model to deter- 
mine if the design of the Pollution Prevention and New Technology research pro- 
gram was conducive to achieving desired environmental outcomes. The pilot was 
successful, and we now employ logic models to clearly identify the outputs of our 
research and their associated near-term outcomes. Logic model techniques are par- 
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ticularly useful for identifying outputs and methods for transferring research results 
to our clients, helping them to achieve environmental outcomes. 

The logic model also emphasizes that there are factors outside the realm of 
science that may help or hinder the success of the program and the accomplishment 
of its results. ORD scientists, EPA program offices, and our State, tribal, and local 
clients each have their respective roles for helping to achieve environmental out- 
comes. In light of this, ORD believes that research programs are most appropriately 
evaluated with respect to the soundness of the research strategy, the significance 
of the research findings, and the usefulness of the resulting scientific tools or poli- 
cies for their intended applications. We also believe there is an important role for 
independent, expert peer review for accomplishing such evaluations. 

ORD is moving forward with its plans to conduct reviews of its research programs 
by external independent experts. These expert panels will review our research in 
accordance with the Administration’s investment criteria for research and develop- 
ment; namely, quality, relevance, and performance. These reviews will provide valu- 
able input for determining that ORD is managing its programs to ensure scientific 
quality, and is providing relevant results for achieving the Agency’s mission. 

It is a challenging task to relate research, especially inherently long-term re- 
search, to specific environmental and public health outcomes. However, as I men- 
tioned earlier, I am committed to moving ORD in that direction. The PARTs con- 
ducted last year have provided up with valuable experience that will help us dem- 
onstrate the value of our programs, and we are working with OMB to develop rec- 
ommendations to improve program performance. 

Science to Achieve Results Research 

Mr. Chairman, your letter of invitation asked me to specifically address the reduc- 
tion of EPA’s STAR grants for research on ecological systems, pollution prevention, 
endocrine disruptors, and mercury. While I will address the specific reductions later 
in my testimony, I want to share at this point some of my thoughts and the 
thoughts of others about our STAR program and how it continues to be a vital part 
of ORD’s research portfolio. 

In 1995, ORD created the Science To Achieve Results extramural research pro- 
gram. This program was created for the purpose of providing ORD swift, flexible ac- 
cess to nationally and internationally acclaimed scientists who could conduct inde- 
pendent and original research to complement the efforts of ORD’s intramural re- 
search program. 

Since the program’s inception, all or parts of the STAR program have been re- 
viewed three times by the EPA SAB and twice by the NRC. These reviews have 
been very favorable, but have also noted areas for improvement. As the NRC is also 
a witness today, I will leave it to them to describe the findings of their 2003 review 
of the STAR program, The Measure of STAR. 

EPA has developed an in-house staff capability to address environmental research 
needs. In some cases, EPA lacks a critical mass of in-house expertise that can de- 
vote itself full-time to new research issues, and the STAR program enables ORD to 
quickly deploy resources to access nationally and internationally acclaimed sci- 
entists to conduct independent and original research where the Agency lacks capac- 
ity or specialized expertise. 

The STAR program remains strong and is aligned to most effectively support 
EPA’s priority research needs. For example, STAR research efforts will be funded 
consistent with previous years’ investments in important areas including children’s 
health, particulate matter, safe food, and drinking water. In those areas where 
STAR will be eliminated in FY 2005 (ecological systems, pollution prevention, endo- 
crine disruptors, and mercury), EPA will continue to conduct in-house research as 
well as look to increase its ongoing research partnerships with university research- 
ers and initiate new ones. STAR currently leverages its resources through joint so- 
licitations with 12 federal and private sector research partners, enabling EPA to en- 
hance its research portfolio by about 30 to 50 additional grants. 

FY 2005 President’s Budget 

The President’s FY 2005 budget request continues the tradition of ORD research 
excellence by emphasizing cutting-edge science and technology, collaboration with 
other agencies, and an orientation on results. 

Mr. Chairman, in your letter of invitation you asked me to identify what research 
would not be done as a result of the proposed reductions in the STAR grants pro- 
gram in the President’s Budget request and the associated impacts. The following 
are areas of decreased STAR research. 
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Ecological Protection Research Grants (-$22. 2M) 

EPA would no longer fund STAR grants in the area of ecological protection, a re- 
duction of about 50 grants. In response to PART findings, EPA is working to develop 
long-term, annual, and efficiency performance measures for the program. Key areas 
of research at academic institutions across the Nation would no longer be conducted, 
affecting Agency efforts to assess ecosystem condition, diagnose ecosystem impair- 
ment, and forecast ecosystem health. 

Hazardous Substance Research Centers (HSRCs) (-$2.25M) 

Eliminate most of the research in the fifth and final year of planned funding for 
the HSRCs, as well as the technical support and outreach efforts of the centers that 
directly support EPA regional, State, and tribal efforts to evaluate and manage risk 
at clean-up sites. 

Mercury Research (~$2.0M) 

Eliminate STAR-supported university research in support of understanding the 
atmospheric processes that affect the transport, transformation, and deposition of 
mercury emissions from natural and anthropogenic sources. 

Endocrine Disruptors (-$4. 7M) 

Eliminate funding for the STAR portion of the Endocrine Disruptors research pro- 
gram. However, the President’s Budget provides a $3.5 million increase for EPA’s 
computational toxicology program, which uses computational chemistry and molec- 
ular biology to more accurately predict health effects from chemicals, thereby im- 
proving linkages between potential exposure and disease. Our computational toxi- 
cology program offers more promising and timely application for our Endocrine 
Disruptors Screening Program. 

Pollution Prevention and New Technologies (-$6.0M) 

Transfer funding of the research program to the Office of Pesticides, Prevention, 
and Toxic Substances pollution prevention program, which the PART analysis has 
shown a reduction in the use of chemicals and pollution. In response to PART find- 
ings, the program is working to develop long-term, annual, and efficiency perform- 
ance measures. 

Conclusion 

By uniquely combining human health and ecological research in one federal agen- 
cy, ORD has made significant contributions to developing a better understanding of 
environmental risks to both human health and ecosystems. The results of this re- 
search have consistently and effectively informed EPA’s environmental decision- 
making, leading to environmental policies based on sound science at the federal, 
State, tribal, and local level. 

The President’s FY 2005 budget request for ORD continues this tradition of excel- 
lence, by emphasizing cutting-edge science and technology, collaboration with other 
agencies, and an orientation on results. 

Thank you. 
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Biography for Paul Gilman 

In April 2002, Dr. Gilman was sworn-in to serve as the Assistant Administrator 
for the Office of Research and Development which is the scientific and technological 
arm of the Environmental Protection Agency. In May 2002, he was appointed the 
Agency Science Advisor. In this capacity, he will be responsible for working across 
the Agency to ensure that the highest quality science is better integrated into the 
Agency’s programs, policies and decisions. 

Before his confirmation, he was Director, Policy Planning for Celera Genomics in 
Rockville, Maryland. Celera Genomics, a bio information and drug discovery com- 
pany, is known for having decoded the human genome. In his position Dr. Gilman 
was responsible for strategic planning for corporate development and communica- 
tions. 

Prior to joining Celera, Dr. Gilman was the Executive Director of the life sciences 
and agriculture divisions of the National Research Council of the National Acad- 
emies of Sciences and Engineering. The National Research Council is the operating 
arm of the National Academies which were chartered to provide independent advice 
to the government in matters of science and engineering. Dr. Gilman’s divisions fo- 
cused on risks to health and the environment, protection and management of biotic 
resources, and practical applications of biology including biotechnology and agri- 
culture. 

Before joining the National Research Council, Gilman was the Associate Director 
of the Office of Management and Budget (OMB) for Natural Resources, Energy, and 
Science. There he coordinated budget formulation, regulatory, and legislative activi- 
ties between agencies such as the Environmental Protection Agency, National 
Science Foundation, Agriculture, and Energy with the Executive Office of the Presi- 
dent. 

Dr. Gilman served as Executive Assistant to the Secretary of Energy for technical 
matters before joining the OMB. His responsibilities included participating in policy 
deliberations and tracking implementation of a variety of programs including the 
Department’s environmental remediation and basic science research. 

Gilman has 13 years of experience working on the staff of the United States Sen- 
ate. He began that time as a Congressional Science Fellow sponsored by the Amer- 
ican Association for the Advancement of Science in the office of Senator Pete V. 
Domenici. Later, as the Staff Director of the Subcommittee on Energy Research and 
Development, he was involved in the passage of the Nuclear Waste Policy Act of 
1982 and oversight of energy technology and environmental research. Later he 
served as the chief-of-staff for Senator Domenici. 

Dr. Gilman matriculated at Kenyon College in Ohio and received his A.B., M.A., 
and Ph.D. degrees in ecology and evolutionary biology from Johns Hopkins Univer- 
sity, Baltimore, Maryland. 

Chairman Ehlers. Thank you. 

Mr. Posner. 

STATEMENT OF MR. PAUL L. POSNER, MANAGING DIRECTOR, 

FEDERAL BUDGET AND INTERGOVERNMENTAL RELATIONS, 

GENERAL ACCOUNTING OFFICE 

Mr. Posner. Thank you, Mr. Chairman. 

My purpose at this hearing is to talk about the report we did on 
the OMB PART and performance budgeting more in general. PART 
is really a new chapter in a long history, some would argue check- 
ered history of performance budgeting, trying to apply performance 
to budgeting which is inherently in political process. One of the les- 
sons we have learned from the failures of PPB, ZBB and OMB is 
it is a fool’s errant. In fact, however, the fact that we keep trying 
suggest we won’t accept failure for an answer because it is too im- 
portant to try to link what we are trying to get out of these pro- 
grams with our resource allocation. This is really the essence of 
what government should be about. GPRA has, in fact, stayed 
around much longer than its predecessors. We just issued a report 
today on that, showing that in fact for 10 years, it has sustained 
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itself and built a credible supply for information and improved 
planning. 

What PART does is take that supply and attempt to more ac- 
tively prompt decision-makers to use the information. Our review 
of the process in fiscal year 2004 suggests that PART succeeded in 
more explicitly developing linkages between performance and budg- 
eting. It more transparently informed budget decisions than we 
have seen before. There is more public transparency about the re- 
sults of the process. Having said that, there is not a direct link 
with funding, nor should there be, because budgeting is an exercise 
in political choice and priorities of which performance is but one 
input. In fact, what we really should expect performance budgeting 
to do is not change the answers in some mechanical way, but to 
change the kind of questions we ask. And in fact, this hearing is 
a very good example of what we are talking about, and I would like 
to see this in other areas. The focus we should have in our budget 
debate is what do we know about programs? What more should we 
know? What should we change in the metrics and the kinds of 
goals that we are setting for these programs? This is actually ex- 
actly what performance budgeting should be all about, what you 
are doing today in this hearing. 

Having said this, doing this is not easy. Developing a bottom line 
of federal programs is not easy. We are not a private sector busi- 
ness, where we have one profit and loss statement. We have mul- 
tiple goals for every program, and it is really difficult to assign a 
single rating to various kinds of balanced portfolios or programs. 
So judgment is always required, regardless of what the number 
says. OMB has tried, in the development of the PART instrument 
and the application, to provide more consistency among its raters, 
among OMB staff and agency people. Some terms will inherently 
be subjective and require judgment; things like what constitutes an 
ambitious performance goal. We found inconsistencies in such 
things as defining what is an outcome versus an output in evalu- 
ating agencies. 

The format of the PART tool itself, with the yes, no kinds of an- 
swers for most questions, force some standardizations for areas 
where ultimately considerable judgment was required to balance 
answers across multiple criteria. The chronic lack of performance 
in evaluation across many programs, which we have long identi- 
fied, remains a considerable barrier in doing this exercise, and one 
thing that we think PART may in fact instigate and instill is a 
greater incentive to get more of this done. Particular challenges, as 
you will note today in research and development, in terms of the 
basic nature of research, makes it more difficult. It takes time to 
really assess what we are getting out of research. Research itself, 
particularly basic research, is an uncertain enterprise. But these 
challenges are being addressed in agencies across the board, and 
we think the National Academy of Sciences has provided useful cri- 
teria. 

One important issue that remains — and when you try to judge 
multiple programs — is defining what a program is. It is arguably 
seemingly something that we should all know, but there is no uni- 
form definition of what a program is, and so this question of what 
is the unit analysis for defining what a program is, is critical in 
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terms of defining what we are getting out of government. This was 
an issue with GPRA, frankly, and the National Academy of Science 
report identified this, and it is clearly an issue with PART. Pro- 
grams were defined one way for GPRA, for example, in more stra- 
tegic ways. Under PART we are redefining programs for budget 
purposes in a more discrete, granular way. These are two different 
competing kinds of accountability mechanisms, and one issue we 
surfaced is the need to better synchronize between the two. 

Ultimately, we recommended continued improvement in the 
guidance and the rating tool by OMB. We recommended a more 
targeted selection process so that, in the future, related programs 
can be grouped together so we can look across different programs 
and see what we are getting. We recommended early consultation 
with the Congress, and most importantly, we suggested that Con- 
gress develop a process to better identify its performance issues 
and oversight priorities to enable it to better communicate with 
OMB and move this process forward. 

Thank you. 

[The prepared statement of Mr. Posner follows:] 

Prepared Statement of Paul L. Posner 

Mr. Chairman and Members of the Subcommittee: 

I am pleased to be here today to discuss performance budgeting and the Office 
of Management and Budget’s (OMB) Program Assessment Rating Tool (PART). 
Since the 1950s, the Federal Government has attempted several government- wide 
initiatives designed to better align spending decisions with expected performance — 
what is commonly referred to as “performance budgeting.” The consensus is that 
prior efforts — including the Hoover Commission, the Planning-Programming-Budg- 
eting-System, Management by Objectives, and Zero-Based Budgeting — did not suc- 
ceed in significantly shifting the focus of the federal budget process from its long- 
standing concentration on the items of government spending to the results of its 
programs. However, the persistent attempts reflect a long-standing interest in link- 
ing resources to results. 

In the 1990s, Congress and the executive branch laid out a statutory and manage- 
ment framework that provides the foundation for strengthening government per- 
formance and accountability, with the Government Performance and Results Act of 
1993 1 (GPRA) as its centerpiece. GPRA is designed to inform congressional and ex- 
ecutive decision-making by providing objective information on the relative effective- 
ness and efficiency of federal programs and spending. A key purpose of the act is 
to create closer and clearer links between the process of allocating scarce resources 
and the expected results to be achieved with those resources. We have learned that 
this type of integration is critical from prior initiatives that failed in part because 
they did not prove to be relevant to budget decision-makers in the executive branch 
or Congress. 2 GPRA requires both a connection to the structures used in congres- 
sional budget presentations and consultation between the executive and legislative 
branches on agency strategic plans; this gives Congress an oversight stake in 
GPRA’s success. 3 

This administration has made the integration of performance and budget informa- 
tion one of five government-wide management priorities under the President’s Man- 
agement Agenda (PMA). 4 Central to this initiative is PART. OMB developed PART 
as a diagnostic tool meant to provide a consistent approach to evaluating federal 
programs and applied it in formulating the President’s fiscal years 2004 and 2005 


4 Pub. L. No. 103-62 (1993). 

2 U.S. General Accounting Office, Performance Budgeting : Past Initiatives Offer Insights for 
GPRA Implementation, GAO/AIMD-97-46 (Washington, D.C.: Mar. 27, 1997). 

3 See Pub. L. No. 103-62 §2 (1993), 5 U.S.C. §306 (2003), and 31 U.S.C. §§1115-1116 (2003). 
4 In addition to budget and performance integration, the other four priorities under the PMA 
are strategic management of human capital, expanded electronic government, improved finan- 
cial performance, and competitive sourcing. 
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budget requests. PART covers four broad topics for all “programs” 5 selected for re- 
view: (1) program purpose and design, (2) strategic planning, (3) program manage- 
ment, and (4) program results (i.e., whether a program is meeting its long-term and 
annual goals) as well as additional questions that are specific to one of seven mecha- 
nisms or approaches used to deliver the program. 6 

GPRA expanded the supply of performance information generated by federal agen- 
cies, although as the PART assessments demonstrate, more must be done to develop 
credible performance information. However, improving the supply of performance in- 
formation is in and of itself insufficient to sustain performance management and 
achieve real improvements in management and program results. Rather, it needs 
to be accompanied by a demand for that information by decision-makers and man- 
agers alike. PART may mark a new chapter in performance-based budgeting by 
more successfully stimulating demand for this information — that is, using the per- 
formance information generated through GPRA’s planning and reporting processes 
to more directly feed into executive branch budgetary decisions. 

My statement today focuses on seven points: 

• PART helped structure OMB’s use of performance information for its internal 
program and budget analysis, made the use of this information more trans- 
parent, and stimulated agency interest in budget and performance integra- 
tion. Moreover, it illustrated the potential to build on GPRA’s foundation to 
more actively promote the use of performance information in budget decisions. 

• The goal of PART is to evaluate programs systematically, consistently, and 
transparently. OMB went to great lengths to encourage consistent application 
of PART in the evaluation of government programs, including pilot testing the 
instrument, issuing detailed guidance, and conducting consistency reviews. 
Although there is undoubtedly room for continued improvement, any tool is 
inherently limited in providing a single performance answer or judgment on 
complex federal programs with multiple goals. 

• Performance measurement challenges in evaluating complex federal programs 
make it difficult to meaningfully interpret a bottom-line rating. The indi- 
vidual section ratings for each PART review provided a better understanding 
of areas needing improvement than the overall rating alone. 

• As is to be expected with any new reform, PART is a work in progress and 
we have noted in our work where OMB might make improvements. Any tool 
that is sophisticated enough to take into account the complexity of the U.S. 
government will require some exercise of judgment. Therefore it is not sur- 
prising that we found some inconsistencies in OMB staff interpreting and ap- 
plying PART. 

• PART provides an opportunity to more efficiently use scarce analytic re- 
sources, to focus decision-makers’ attention on the most pressing policy issues, 
and to consider comparisons and trade-offs among related programs by more 
strategically targeting PART assessments based on such factors as the rel- 
ative priorities, costs, and risks associated with related clusters of programs 
and activities. The first year PART assessments underscored the long-stand- 
ing gaps in performance and evaluation information throughout the Federal 
Government. By reaching agreement on areas in which evaluations are most 
essential, decision-makers can help ensure that limited resources are applied 
wisely. 

• The relationship between PART and its process and the broader GPRA stra- 
tegic planning process is still evolving. Although PART can stimulate discus- 
sion on program-specific performance measurement issues, it is not a sub- 
stitute for GPRA’s strategic, longer-term focus on thematic goals and 
department- and government-wide crosscutting comparisons. PART and 
GPRA serve different but complementary needs, so a strategy for integrating 
the two could help strengthen both. 

• Federal programs are designed and implemented in dynamic environments 
where competing program priorities and stakeholders’ needs must be bal- 
anced continually and new needs must be addressed. While PART clearly 


5 There is no standard definition for the term “program.” For purposes of PART, OMB de- 
scribed the unit of analysis (program) as (1) an activity or set of activities clearly recognized 
as a program by the public, OMB, or Congress; (2) having a discrete level of funding clearly 
associated with it; and (3) corresponding to the level at which budget decisions are made. 

6 The seven major categories are competitive grants, block/formula grants, capital assets and 
service acquisition programs, credit programs, regulatory-based programs, direct federal pro- 
grams, and research and development programs. 
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serves the needs of OMB in budget formulation, questions remain about 
whether it serves the various needs of other key stakeholders. If the Presi- 
dent or OMB wants PART and its results to be considered in the congres- 
sional debate, it will be important for OMB to (1) involve congressional stake- 
holders early in providing input on the focus of the assessments; (2) clarify 
any significant limitations in the assessments as well as the underlying per- 
formance information; and (3) initiate discussions with key congressional com- 
mittees about how they can best take advantage of and leverage PART infor- 
mation in congressional authorization, appropriations, and oversight proc- 
esses. Moreover, Congress needs to consider ways it can articulate its over- 
sight priorities and performance agenda. 

My statement is based on our recently published report on OMB’s PART 7 in 
which we reviewed the first year of the PART process — fiscal year 2004 — and 
changes in the PART process initiated for fiscal year 2005. We have not reviewed 
or analyzed the PART results for the fiscal year 2005 budget request. For this testi- 
mony, this subcommittee asked us to discuss our overall findings and recommenda- 
tions concerning PART to help frame today’s hearing. We conducted our work in ac- 
cordance with generally accepted government auditing standards. 

Strengths and Weaknesses of PART in Its First Year of Implementation 

Through its development and use of PART, OMB has more explicitly infused per- 
formance information into the budget formulation process; increased the attention 
paid to performance information and program evaluations; and ultimately, we hope, 
increased the value of this information to decision-makers and other stakeholders. 
By linking performance information to the budget process, OMB has provided agen- 
cies with a powerful incentive for improving both the quality and availability of per- 
formance information. The level of effort and involvement by senior OMB officials 
and staff clearly signals the importance of this strategy in meeting the priorities 
outlined in the PMA. OMB should be credited with opening up for scrutiny — and 
potential criticism — its review of key areas of federal program performance and then 
making its assessments available to a potentially wider audience through its Web 
site. 

As OMB and others recognize, performance is not the only factor in funding deci- 
sions. Determining priorities — including funding priorities — is a function of com- 
peting values and interests. Accordingly, we found that while PART scores were 
generally positively related to proposed funding changes in discretionary programs, 
the scores did not automatically determine funding changes. That is, for some pro- 
grams rated “effective” or “moderately effective” OMB recommended funding de- 
creases, while for several programs judged to be “ineffective” OMB recommended ad- 
ditional funding in the President’s budget request with which to implement changes. 
In fact, the more important role of PART was not its use in making resource deci- 
sions, but in its support for recommendations to improve program design, assess- 
ment, and management. Our analysis of the fiscal year 2004 PART found that 82 
percent of the recommendations addressed program assessment, design, and man- 
agement issues; only 18 percent of the recommendations had a direct link to funding 
matters. 8 

OMB’s ability to use PART to identify and address future program improvements 
and measure progress — a major purpose of PART — depends on its ability to oversee 
the implementation of PART recommendations. As OMB has recognized, following 
through on these recommendations is essential for improving program performance 
and ensuring accountability. Currently, OMB plans to assess an additional 20 per- 
cent of all federal programs annually. As the number of recommendations from pre- 
vious years’ evaluations grows, a system for monitoring their implementation will 
become more critical. However, OMB does not have a centralized system to oversee 
the implementation of such recommendations or evaluate their effectiveness. 

The goal of PART is to evaluate programs systematically, consistently, and trans- 
parently. OMB went to great lengths to encourage consistent application of PART 
in the evaluation of government programs, including pilot testing the instrument, 
issuing detailed guidance, and conducting consistency reviews. Although there is un- 
doubtedly room for continued improvement, any tool is inherently limited in pro- 
viding a single performance answer or judgment on complex federal programs with 
multiple goals. 


7 U.S. General Accounting Office, Performance Budgeting .-Observations on the Use of OMB’s 
Program Assessment Rating Tool for the Fiscal Year 2004 Budget, GAO— 04-174 (Washington, 
D.C.: Jan. 30, 2004). 

8 The 234 programs assessed for fiscal year 2004 contained a total of 612 recommendations. 
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OMB recognized the complexity inherent in evaluating federal programs by dif- 
ferentiating its rating tool for seven mechanisms or approaches used to deliver serv- 
ices, ranging from block grants to research and development. However, judgment is 
involved in classifying programs by these categories since many programs fit into 
more than one of these groupings. OMB guidance, for instance, acknowledges that 
some research and development programs can also be evaluated as competitive 
grants and capital assets. 

Performance measurement challenges in evaluating complex federal programs 
make it difficult to meaningfully interpret a bottom-line rating. OMB published both 
a single, bottom-line rating for PART results and individual section scores. It is 
these latter scores that are potentially more useful for identifying information gaps 
and program weaknesses. For example, in the fiscal year 2004 PART, one program 
that was rated “adequate” overall got high scores for purpose (80 percent) and plan- 
ning (100 percent), but poor scores in being able to show results (39 percent) and 
in program management (46 percent). In a case like this, the individual section rat- 
ings provided a better understanding of areas needing improvement than the overall 
rating alone. In addition, bottom-line ratings may force raters to choose among sev- 
eral important but disparate goals and encourage a determination of program effec- 
tiveness even when performance data are unavailable, the quality of those data is 
uneven, or they convey a mixed message on performance. 

Any tool that is sophisticated enough to take into account the complexity of the 
U.S. government will always require some interpretation and judgment. Therefore 
it is not surprising that OMB staff were not fully consistent in interpreting complex 
questions about agency goals and results. Many PART questions contain subjective 
terms that are open to interpretation. Examples include terminology such as “ambi- 
tious” in describing sought-after performance measures. Because the appropriate- 
ness of a performance measure depends on the program’s purpose, and because pro- 
gram purposes can vary immensely, an ambitious goal for one program might be 
unrealistic for a similar but more narrowly defined program. Without further guid- 
ance, it is unclear how OMB staff can be expected to be consistent. 

We also found inconsistencies in how the definition of acceptable performance 
measures was applied. Our review of the fiscal year 2004 PART surfaced several 
instances in which OMB staff inconsistently defined appropriate measures — outcome 
versus output — for programs. Agency officials also told us that OMB staff used dif- 
ferent standards to define measures as outcome-oriented. Outputs are the products 
and services delivered by the program whereas outcomes refer to the results of out- 
puts. For example, in the employment and training area, OMB accepted short-term 
outcomes, such as obtaining high school diplomas or employment, as a proxy for 
long-term goals for the Department of Health and Human Services’ Refugee Assist- 
ance program, which aims to help refugees attain economic self-sufficiency as soon 
as possible. However, OMB did not accept the same employment measure as a proxy 
for long-term goals for the Department of Education’s Vocational Rehabilitation pro- 
gram because it had not set long-term targets beyond a couple of years. In other 
words, although neither program contained long-term outcomes, such as partici- 
pants gaining economic self-sufficiency, OMB accepted short-term outcomes in one 
instance but not the other. 

The yes/no format employed throughout most of the PART questionnaire resulted 
in oversimplified answers to some questions. Although OMB believes it helped 
standardization, the yes/no format was particularly troublesome for questions con- 
taining multiple criteria for a “yes” answer. Agency officials have commented that 
the yes/no format can oversimplify reality, in which progress in planning, manage- 
ment, or results is more likely to resemble a continuum than an on/off switch. Our 
review of the fiscal year 2004 PART found several instances in which some OMB 
staff gave a “yes” answer for successfully achieving some but not all of the multiple 
criteria, while others gave a “no” answer when presented with a similar situation. 
For example, OMB judged the Department of the Interior’s (DOI) Water Reuse and 
Recycling program “no” on whether a program has a limited number of ambitious, 
long-term performance goals, noting that although DOI set a long-term goal of 
500,000 acre-feet per year of reclaimed water, it failed to establish a time frame for 
when it would reach the target. However, OMB judged the Department of Agri- 
culture’s and DOI’s Wildland Fire programs “yes” on this question even though the 
programs’ long-term goals of improved conditions in high-priority forest acres are 
not accompanied by specific time frames. 

The lack of program performance information also creates challenges in effectively 
assessing program performance. According to OMB, about half of the programs as- 
sessed for fiscal year 2004 lacked “specific, ambitious long-term performance goals 
that focus on outcomes” and nearly 40 percent lacked sufficient “independent, qual- 
ity evaluations.” Nearly 50 percent of programs assessed for fiscal year 2004 re- 
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ceived ratings of “results not demonstrated” because OMB decided that program 
performance information, performance goals, or both were insufficient or inad- 
equate. While the validity of these assessments may be subject to interpretation and 
debate, our previous work 9 has raised concerns about the capacity of federal agen- 
cies to produce evaluations of program effectiveness as well as credible data. 

In our report on PART, we note that several factors have limited the availability 
of performance data and evaluations of federal programs, including the lack of stat- 
utory mandates and funding to support data collection and analysis. Our work has 
recognized that research programs pose particular and long-standing challenges for 
performance assessments and evaluations. 10 For instance, in both applied and basic 
research, projects take several years to complete and require more time before their 
meaning for the field can be adequately understood and captured in performance re- 
porting systems. These challenges can and have been addressed by federal and pri- 
vate research organizations. One evaluation approach we have identified in our re- 
view of leading practices is the use of peer review to evaluate the quality of research 
outcomes. 11 For example, the National Science Foundation (NSF) convenes panels 
of independent experts as external advisers — a Committee of Visitors (COV) — to 
peer review the technical and managerial stewardship of a specific program or clus- 
ter of programs periodically. The COV compares research plans with progress made, 
and evaluates outcomes to determine whether the research contributes to NSF mis- 
sion and goals. 

The Relationship between GPRA and PART 

PART was designed for and is used in the executive branch budget preparation 
and review process. As a result, the goals and measures used in PART must meet 
OMB’s needs. By comparison, GPRA — the current statutory framework for strategic 
planning and reporting — is a broader process involving the development of strategic 
and performance goals and objectives to be reported in strategic and annual plans 
and reports. OMB said that GPRA plans were organized at too high a level to be 
meaningful for program-level budget analysis and management review. OMB ac- 
knowledges that GPRA was the starting point for PART, but as I will explain, it 
appears that OMB’s emphasis is shifting such that over time the performance meas- 
ures developed for PART and used in the budget process may also come to drive 
agencies’ strategic planning processes. 

The fiscal year 2004 PART process came to be a parallel competing structure to 
the GPRA framework as a result of OMB’s desire to collect performance data that 
better align with budget decision units. OMB’s most recent Circular A-ll guidance 
clearly requires both that each agency submit a performance budget for fiscal year 
2005 and that this should replace the annual GPRA performance plan. 12 These per- 
formance budgets are to include information from the PART assessments, where 
available, including all performance goals used in the assessment of program per- 
formance done under the PART process. Until all programs have been assessed 
using PART, the performance budget will also include performance goals for agency 
programs that have not yet been assessed. OMB’s movement from GPRA to PART 
is further evident in the fiscal year 2005 PART guidance stating that while existing 
GPRA performance goals may be a starting point during the development of PART 
performance goals, the GPRA goals in agency GPRA documents are to be revised, 
as needed, to reflect OMB’s instructions for developing the PART performance goals. 
Lastly, this same guidance states that GPRA plans should be revised to include any 
new performance measures used in PART and that unnecessary measures should 
be deleted from GPRA plans. In its comments to another recently issued GAO re- 
port, OMB stated that it will revise its guidance for both GPRA and PART to clarify 
the integrated and complementary relationship between the two initiatives. 13 

Although there is potential for complementary approaches to GPRA and PART, 
the following examples clearly illustrate the importance of carefully considering the 
implications of selecting a unit of analysis, including its impact on the availability 
of performance data. They also reveal some of the unresolved tensions between the 
President’s budget and performance initiative — a detailed budget perspective — and 


9 U.S. General Accounting Office, Program Evaluation: Agencies Challenged by New Demand 
for Information on Program Results, GAO/GGD— 98-53 (Washington, D.C.: Apr. 24, 1998). 

10 U.S. General Accounting Office, Transportation Research: Actions Needed to Improve Coordi- 
nation and Evaluation of Research, GAO-03-500 (Washington, D.C.: May 1, 2003). 

1:L U.S. General Accounting Office, Program Evaluation: An Evaluation Culture and Collabo- 
rative Partnerships Help Build Agency Capacity, GAO-03—454 (Washington, D.C.: May 2, 2003). 

12 OMB Circular A— 11, Preparation, Submission, and Execution of the Budget. 

13 U.S. General Accounting Office, Results-Oriented Government: GPRA Has Established a 
Solid Foundation for Achieving Greater Results, GAO— 04-38 (Washington, D.C.: March 10, 
2004). 
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GPRA — a more strategic planning view. Experience with PART highlighted the fact 
that defining a “unit of analysis” useful for both program-level budget analysis and 
agency planning purposes can be difficult. For example, disaggregating programs for 
PART purposes could ignore the interdependence of programs recognized by GPRA 
by artificially isolating programs from the larger contexts in which they operate. 
Agency officials described one program assessed with the fiscal year 2004 PART — 
Projects for Assistance in Transition from Homelessness — that was aimed at a spe- 
cific aspect of homelessness, that is, referring persons with emergency needs to 
other agencies for housing and needed services. OMB staff wanted the agency to 
produce long-term outcome measures for this program to support the PART review 
process. Agency officials argued that chronically homeless people require many serv- 
ices and that this federal program often supports only some of the services needed 
at the initial stages of intervention. GPRA — with its focus on assessing the relative 
contributions of related programs to broader goals — is better designed to consider 
crosscutting strategies to achieve common goals. Federal programs cannot be as- 
sessed in isolation. Performance also needs to be examined from an integrated, stra- 
tegic perspective. 

One way of improving the links between PART and GPRA would be to develop 
a more strategic approach to selecting and prioritizing areas for assessment under 
the PART process. Targeting PART assessments based on such factors as the rel- 
ative priorities, costs, and risks associated with related clusters of programs and ac- 
tivities addressing common strategic and performance goals not only could help ra- 
tion scarce analytic resources but also could focus decision-makers’ attention on the 
most pressing policy and program issues. Moreover, such an approach could facili- 
tate the use of PART assessments to review the relative contributions of similar pro- 
grams to common or crosscutting goals and outcomes established through the GPRA 
process. 

The Importance of Congressional and Other Stakeholder Involvement 

We have previously reported 14 that stakeholder involvement appears critical for 
getting consensus on goals and measures. In fact, GPRA requires agencies to consult 
with Congress and solicit the views of other stakeholders as they develop their stra- 
tegic plans. 15 Stakeholder involvement can be particularly important for federal 
agencies because they operate in a complex political environment in which legisla- 
tive mandates are often broadly stated and some stakeholders may strongly disagree 
about the agency’s mission and goals. 

The relationship between PART and its process and the broader GPRA strategic 
planning process is still evolving. As part of the executive branch budget formula- 
tion process, PART must clearly serve the President’s interests. Some tension about 
the amount of stakeholder involvement in the internal deliberations surrounding 
the development of PART measures and the broader consultations more common to 
the GPRA strategic planning process is inevitable. Compared to the relatively open- 
ended GPRA process, any budget formulation process is likely to seem closed. 

Yet, we must ask whether the broad range of congressional officials with a stake 
in how programs perform will use PART assessments unless they believe the re- 
views reflect a consensus about performance goals among a community of interests, 
target performance issues that are important to them as well as the administration, 
and are based on an evaluation process in which they have confidence. Similarly, 
the measures used to demonstrate progress toward a goal, no matter how worth- 
while, cannot serve the interests of a single stakeholder or purpose without poten- 
tially discouraging use of this information by others. 

Congress has a number of opportunities to provide its perspective on performance 
issues and performance goals, such as when it establishes or reauthorizes a new 
program, during the annual appropriations process, and in its oversight of federal 
operations. In fact, these processes already reflect GPRA’s influence. Reviews of lan- 
guage in public laws and committee reports show an increasing number of ref- 
erences to GPRA-related provisions. What is missing is a mechanism to systemati- 
cally coordinate a congressional perspective and promote a dialogue between Con- 
gress and the President in the PART review process. 

In our report, we have suggested steps for both OMB and the Congress to take 
to strengthen the dialogue between executive officials and congressional stake- 
holders. We have recommended that OMB reach out to key congressional commit- 
tees early in the PART selection process to gain insight about which program areas 
and performance issues congressional officials believe warrant PART review. Engag- 


14 U.S. General Accounting Office, Agencies’ Strategic Plans Under GPRA: Key Questions to 
Facilitate Congressional Review (Version 1), GAO/GGD-IO.1.16 (Washington, D.C.: May 1997). 
15 5 U.S.C. §306(d) (2003). 
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ing Congress early in the process may help target reviews with an eye toward those 
areas most likely to be on the agenda of Congress, thereby better ensuring the use 
of performance assessments in resource allocation processes throughout government. 
We have also suggested that Congress consider the need to develop a more system- 
atic vehicle for communicating its top performance concerns and priorities; develop 
a more structured oversight agenda to prompt a more coordinated congressional per- 
spective on crosscutting performance issues; and use this agenda to inform its au- 
thorization, appropriations, and oversight processes. 

Concluding Observations 

The PART process is the latest initiative in a long-standing series of reforms un- 
dertaken to improve the link between performance information and budget deci- 
sions. Although each of the initiatives of the past appears to have met with an early 
demise, in fact, subsequent reforms were strengthened by building on the legacy left 
by their predecessors. Prior reforms often failed because they were not relevant to 
resource allocation and other decision-making processes, thereby eroding the incen- 
tives for federal agencies to improve their planning, data, and evaluations. 

Unlike many of those past initiatives, GPRA has been sustained since its passage 
10 years ago, and evidence exists that it has become more relevant than its prede- 
cessors. PART offers the potential to build on the infrastructure of performance 
plans and information ushered in by GPRA and the law’s intent to promote the use 
of these plans in resource allocation decision-making. GPRA improved the supply of 
plans and information, while PART can prompt greater demand for this information 
by decision-makers. Enhancing interest and use may bring about greater incentives 
for agencies to devote scarce resources to improving their information and evalua- 
tions of federal programs as well. 

Increasing the use and usefulness of performance data is not only important to 
sustain performance management reforms, but to improve the processes of decision- 
making and governance. Many in the United States believe there is a need to estab- 
lish a comprehensive portfolio of key national performance indicators. This will raise 
complex issues ranging from agreement on performance areas and indicators to get- 
ting and sharing reliable information for public planning, decision-making, and ac- 
countability. In this regard, the entire agenda of management reform at the federal 
level has been focused on shifting the attention of decision-makers and agency man- 
agement from process to results. Although PART is based on changing the orienta- 
tion of budgeting, other initiatives championed by Congress and embodied in the 
PMA are also devoted to improving the accountability for performance goals in agen- 
cy human capital management, financial management, competitive sourcing, and 
other key management areas. 

In particular, we have reported that human capital — or people — is at the center 
of any serious change management initiative. Thus, strategic human capital man- 
agement is at the heart of government transformation. High-performing organiza- 
tions strengthen the alignment of their GPRA strategic and performance goals with 
their daily operations. In that regard, performance management systems can be a 
vital tool for aligning an organization’s operations with individual day-to-day activi- 
ties, but they are currently largely unused. As we move forward to strengthen gov- 
ernment performance and accountability, effective performance management sys- 
tems can be a strategic tool to drive internal change and achieve desired results. 

The question now is how to enhance the credibility and use of the PART process 
as a tool to focus decisions on performance. In our report, we make seven rec- 
ommendations to OMB and a suggestion to Congress to better support the kind of 
collaborative approach to performance budgeting that very well may be essential in 
a separation of powers system like ours. Our suggestions cover several key issues 
that need to be addressed to strengthen and help sustain the PART process. We rec- 
ommend that the OMB Director take the following actions: 

• Centrally monitor agency implementation and progress on PART rec- 
ommendations and report such progress in OMB’s budget submission to Con- 
gress. Government-wide councils may be effective vehicles for assisting OMB 
in these efforts. 

• Continue to improve the PART guidance by (1) expanding the discussion of 
how the unit of analysis is to be determined to include trade-offs made when 
defining a unit of analysis, implications of how the unit of analysis is defined, 
or both; (2) clarifying when output versus outcome measures are acceptable; 
and (3) better defining an “independent, quality evaluation.” 

• Clarify OMB’s expectations to agencies regarding the allocation of scarce eval- 
uation resources among programs, the timing of such evaluations, as well as 
the evaluation strategies it wants for PART, and consider using internal 
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agency evaluations as evidence on a case-by-case basis — whether conducted 
by agencies, contractors, or other parties. 

• Reconsider plans for 100 percent coverage of federal programs and, instead, 
target for review a significant percentage of major and meaningful govern- 
ment programs based on such factors as the relative priorities, costs, and 
risks associated with related clusters of programs and activities. 

• Maximize the opportunity to review similar programs or activities in the 
same year to facilitate comparisons and trade-offs. 

• Attempt to generate, early in the PART process, an ongoing, meaningful dia- 
logue with congressional appropriations, authorization, and oversight commit- 
tees about what they consider to be the most important performance issues 
and program areas that warrant review. 

• Seek to achieve the greatest benefit from both GPRA and PART by articu- 
lating and implementing an integrated, complementary relationship between 
the two. 

In its comments on our report, OMB outlined actions it is taking to address sev- 
eral of these recommendations, including refining the process for monitoring agen- 
cies’ progress in implementing the PART recommendations, seeking opportunities 
for dialogue with Congress on agencies’ performance, and continuing to improve ex- 
ecutive branch implementation of GPRA plans and reports. 

Our recommendations to OMB are partly directed at fortifying and enhancing the 
credibility of PART itself and the underlying data used to make the judgments. De- 
cision makers across government are more likely to rely on PART data and assess- 
ments if the underlying information and the rating process are perceived as being 
credible, systematic, and consistent. Enhanced OMB guidance and improved strate- 
gies for obtaining and evaluating program performance data are vital elements. 

The PART process can be made more sustainable if the use of analytic resources 
at OMB and the agencies is rationalized by reconsidering the goal of 100 percent 
coverage of all federal programs. Instead, we suggest a more strategic approach to 
target assessments on related clusters of programs and activities. A more targeted 
approach stands a better chance of capturing the interest of decision-makers 
throughout the process by focusing their attention on the most pressing policy and 
program issues and on how related programs and tools affect broader crosscutting 
outcomes and goals. Unfortunately, the government-wide performance plan required 
by GPRA has never been engaged to drive budgeting in this way. 

Improving the integration of inherently separate but interrelated strategic plan- 
ning and performance budgeting processes can help support a more strategic focus 
for PART assessments. GPRA’s strategic planning goals could be used to anchor the 
selection and review of programs by providing a foundation to assess the relative 
contribution of related programs and tools to broader performance goals and out- 
comes. 

Finally, refining the PART questionnaire and review process and improving the 
quality of data are important, but the question of whose interests drive the process 
is perhaps paramount in our system. Ultimately, the impact of PART on decision- 
making will be a function not only of the President’s decisions, but of congressional 
decisions as well. 

Much is at stake in the development of a collaborative performance budgeting 
process. Not only might the PART reviews ultimately come to be disregarded absent 
congressional involvement, but more important, Congress will lose an opportunity 
to use the PART process to improve its own decision-making and oversight proc- 
esses. 

This is an opportune time for the executive branch and Congress to carefully con- 
sider how agencies and committees can best take advantage of and leverage the new 
information and perspectives coming from the reform agenda under way in the exec- 
utive branch. Ultimately, the specific approach or process is not important. We face 
a long-term fiscal imbalance, which will require us to re-examine our existing poli- 
cies and programs. It is all too easy to accept “the base” as given and to subject 
only new proposals to scrutiny and analysis. The norm should be to reconsider the 
relevance or “fit” of any federal program, policy, or activity in today’s world and for 
the future. 

Mr. Chairman, this concludes my prepared statement. I would be pleased to an- 
swer any questions you or the other Members of the Subcommittee may have at this 
time. 
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at (202) 512-9573 or posnerp@gao.gov. 


What GAO Found 

PART helped structure OMB’s use of performance information for internal 
program and budget analysis and stimulated agency interest in budget and 
performance integration. Moreover, it illustrated the potential to build on 
GPRA’s foundation to more actively promote the use of performance 
information in budget decisions. OMB deserves credit for inviting scrutiny of its 
federal program performance reviews and sharing them on its Web site. 

The goal of PART is to evaluate programs systematically, consistently, and 
transparently. OMB went to great lengths to encourage consistent application of 
PART in the evaluation of government programs, including pilot testing the 
instrument, issuing detailed guidance, and conducting consistency reviews. 
Although there is undoubtedly room for continued improvement, any tool is 
inherently limited in providing a single performance answer or judgment on 
complex federal programs with multiple goals. 

Performance measurement challenges in evaluating complex federal programs 
make it difficult to meaningfully interpret a single bottom-line rating. The 
individual section ratings for each PART review provided a better understanding 
of areas needing improvement than the overall rating alone. Moreover, any tool 
that is sophisticated enough to take into account the complexity of the U.S. 
government will always require some interpretation and judgment. Therefore it 
is not surprising that OMB staff were not fully consistent in interpreting complex 
questions about agency goals and results. 

The lack of program performance information at the agency level also creates 
challenges in effectively measuring program performance. PART provides an 
opportunity to consider strategically targeting the assessments on groups of 
related programs contributing to common outcomes to more efficiently use 
scarce analytic resources and focus decision makers’ attention on the most 
pressing performance issues cutting across individual programs and agencies. 

The relationship between PART and the broader GPRA strategic planning 
process is still evolving and highlights the critical importance of defining the unit 
of analysis for program evaluation. Although PART can stimulate discussion on 
program-specific performance measurement issues, it is not a substitute for 
GPRA’s strategic, longer-term focus on thematic goals, and department- and 
govemmentwide crosscutting comparisons. 

PART clearly serves OMB’s needs, but questions remain about whether it serves 
the various needs of other key stakeholders. If PART results are to be 
considered in the congressional debate, it will be important for OMB to 
(1) involve congressional stakeholders early in providing input on the focus of 
the assessments; (2) clarify any significant limitations in the assessments and 
underlying performance information; and (3) initiate discussions with key 
congressional committees about how they can best leverage PART information 
in congressional authorization, appropriations, and oversight processes. 

United States General Accounting Office 
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Chairman Ehlers. Dr. Matanoski. 

STATEMENT OF DR. GENEVIEVE MATANOSKI, PROFESSOR, DE- 
PARTMENT OF EPIDEMIOLOGY, JOHNS HOPKINS UNIVER- 
SITY; CHAIR, EPA SCIENCE ADVISORY BOARD REVIEW OF 

EPA’S FISCAL YEAR 2005 BUDGET; FORMER CHAIR, EPA 

SCIENCE ADVISORY BOARD 

Dr. Matanoski. Good morning, Mr. Chairman and Members of 
the Subcommittee on Environment, Technology and Standards. 

My name is Genevieve Matanoski. I am honored to appear before 
you today to present the views of the Science Advisory Board for 
EPA regarding their Fiscal Year 2005 Science and Research Budget 
Request. The Board will complete its full report by March 19, 2004, 
and with the permission of the Chairman, we will submit that re- 
port for the record then. 

Over the last year, the Board, under the leadership of its Chair- 
man, Dr. William Glaze, has reorganized to make its advisory func- 
tion as apparent as its peer-review function. With this reorganiza- 
tion, the Board moved the EPA Science and Research Budget Re- 
view from a small standing committee to the entire larger Board, 
and this has provided great visibility to the review, and it also has 
increased the expertise available to conduct the evaluation. This 
Board, in the past years, has noted its deep concern over the con- 
stant erosion of EPA’s research budget programs. The fiscal year 
2005 budget we now observe not only continued the erosion but ac- 
tually cut funding for research. This continuous decrease in re- 
search funding severely constrains the ability of the agency to pro- 
vide the necessary science to guide informed decision-making, and 
to assist in reaching the Nation’s goals for human health and envi- 
ronmental risk reduction. 

In particular, the Board notes, as did the Chairman, substantial 
cuts in their fiscal year 2005 budget request for EPA’s Science to 
Achieve Results program, STAR. STAR is recognized by this Board 
as the science program of major importance to the agency. That 
view is consistent with the views of the National Academy of 
Science in its review of STAR. STAR provides many benefits to 
EPA, one of which is the necessary flexibility to obtain critical sci- 
entific expertise in a wide range of disciplines that are essential in 
addressing emerging issues that are outside EPA’s current areas of 
expertise. EPA could never maintain the same large base of sci- 
entific expertise that is available on an as-needed basis to carry out 
specific research. STAR also enhances EPA’s collaborations with 
outside researchers and academic institutions, and in the process 
actually stimulates additional resources for the agency and its 
science needs. 

STAR also benefits and strengthens scientific research through- 
out the United States by providing training for graduate students 
who will reinforce the declining base of engineers and scientists in 
the United States. Even though STAR is largely focused on EPA’s 
core research, it has begun to accrue a record of early success. Seri- 
ous budget cuts in STAR are as follows. Ecosystems protection re- 
search got a $22 million loss, with a loss of some 50 STAR grants 
based in many states. That is a serious reduction. Endocrine 
disruptors’ research was cut by five million. Pollution prevention 
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research, which is — a focus of that research is to avoid future prob- 
lems. That was cut by five million. And mercury research was cut 
by another two million. Even the STAR Graduate Fellowship Pro- 
gram, despite its increases, is still $4 million short of what Con- 
gress recommended for fiscal year 2003. 

The Board believes that these cuts will have a negative impact 
on the balance research portfolio that EPA has developed over the 
last decade. In that time, EPA has developed a program that bal- 
ances its problem-driving, short-term applied research, whatever 
you want to call it, with its core long-term basic research. Here, the 
STAR program helps EPA balance its internal and its extramural 
research portfolios. The result is that science from many different 
institutions is integrated into a total research program that com- 
pliments the scientific niche filled by EPA’s own excellent sci- 
entists. This provides a more nimble resource that is available to 
work on existing and emerging environmental issues than would be 
available with only an intramural program. 

Another surprising change as already noted in the budget was 
EPA’s decrease in its commitment to Homeland Security. The 
Board believes that EPA must play a continuing role in Homeland 
Security in several areas, and one of these is building contamina- 
tion, which was cut. 

I have run out of time. There are several other issues that we 
have in our written testimony, one regarding PART and the other 
regarding the future of research at EPA. 

[The prepared statement of Dr. Matanoski follows:] 

Prepared Statement of Genevieve Matanoski 

Good morning Mr. Chairman and Members of the Subcommittee on Environment, 
Technology, and Standards. My name is Genevieve Matanoski. I am honored to ap- 
pear before you today to present the conclusions of the EPA Science Advisory Board 
(SAB) on the EPA’s FY 2005 Science and Research Budget Request. My comments 
today will summarize the views of the Board about that request for funding EPA’s 
science and research programs during FY 2005. The Board will complete its full re- 
port on this issue by March 19, 2004 and with the permission of the Chairman, will 
submit that report for the record then. 

1. Introductory Remarks 

Over the last year the Board, under the leadership of its Chairman, Dr. William 
Glaze, has reorganized to make its advisory function as apparent as its peer review 
function. To take advantage of the benefits of this reorganization, the Board moved 
the EPA Science and Research advisory function from a small standing committee 
to the larger Board. This enhances the visibility of the review and increases the ex- 
pertise available to conduct the evaluation. We will also be working with EPA to 
obtain information on the Agency’s science and research programs on a systematic 
and continuous basis. The Agency now presents information to the Board in ways 
that correlate with their new Strategic Plan and which will ultimately include all 
science and research programs regardless of their funding source or where they are 
conducted within EPA. 

2. Specific Comments on EPA’s FY 2005 Science and Research Program and 

Budgets 

a. Erosion of the EPA Research Budget 

First, I want to mention that in past years, the Board has noted its deep concern 
with the constant erosion of EPA’s research program budgets. For the FY 2005 
budget, we now observe not just continued erosion, but a substantial cut to research 
funding. This continuous decrease in science and research funding severely con- 
strains the ability of the Agency to provide the necessary science to guide informed 
decision-making and to assist in reaching the Nation’s goals for human health and 
environmental risk reduction. The EPA has an outstanding group of scientists who 
conduct excellent basic and applied research focused on reducing both recognized 
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problems such as air pollution, and the continually emerging problems arising from 
the past misuse of our environment. 

b. Cuts to the STAR Program 

In particular, the Board notes the substantial cuts in the FY 2005 budget request 
for EPA’s Science to Achieve Results program (STAR). STAR is recognized by this 
Board as a science program of major importance to the Agency. That view is con- 
sistent with the views of the National Academy of Science in its 2003 review enti- 
tled “The Measure of STAR.” STAR provides many benefits to EPA, including the 
necessary flexibility to obtain critical scientific expertise in the wide range of dis- 
ciplines that are essential in addressing emerging issues that are outside EPA’s cur- 
rent areas of expertise. EPA could never maintain the same large base of scientific 
expertise that is available on an as needed basis to carry out specific research. 
STAR enhances EPA’s collaboration with outside researchers and academic institu- 
tions and in the process actually stimulates additional resources for Agency science 
needs. STAR also benefits and strengthens scientific research throughout the United 
States by providing training for graduate students who will reinforce the declining 
base of engineers and scientists in the U.S. Even though STAR is largely focused 
on EPA’s core research, it has begun to accrue a record of early success. Evidence 
of this can be found in the NAS’s “Measure of STAR” report in which it states that 
STAR research on endocrine disruptors, and ecologic assessment have already re- 
sulted in “. . .peer-reviewed publications that are of immediate use in under- 
standing causes, exposures, and effects of environmental pollution.” By any meas- 
ure, STAR is an excellent investment. 

To emphasize the seriousness of this situation, we note a number of cuts to STAR 
research which are a part of the FY 2005 budget request: 

i. Ecosystems Protection Research is reduced by over $22 million with a loss 
of some 50 STAR grants based in many states. The Board is particularly 
concerned about this cut given the critical need for ecosystems research, 
which the Board feels is generally under-funded across EPA. 

ii. Endocrine Disruptors Research is reduced by about $5 million. This is an 
area of research that investigates the effects that could be associated with 
use of many chemicals used in large quantities in our society, 

iii. Pollution Prevention Research is reduced by $5 million even though the 
focus of this research is on avoiding future problems and reducing the ex- 
pensive cleanup costs that we face today, 

iv. Mercury Research is reduced by $2 million, just at the time when more in- 
formation is needed on this ubiquitous contaminant. 

In addition, even though the STAR Graduate Fellowship program increases by 
$1.2 million over the FY 2004 level, it is still nearly $4 million below the level en- 
acted by Congress for FY 2003 ($9.8 million). This program’s aim is to educate the 
future environmental scientists that will be needed to replace the currently aging 
population of such scientists. Thus, adequate funding of this program continues to 
be essential. 

The Board believes that these cuts will have a negative impact on the balanced 
research portfolio that EPA, especially ORD, has developed over the last decade. In 
that time, EPA has developed a program that balances its problem-driven (shorter- 
term, applied) research with its core (longer-term, basic) research. Though compo- 
nents of the core program are not always easy to identify in the budget, EPA ap- 
pears to have a balanced research program, in this regard, with about half being 
“core” and half being “problem-driven.” This seems to be appropriate. 

Further, the STAR program helps EPA balance its internal research portfolio with 
its extramural research portfolio. The result is that science from many different in- 
stitutions (government, academia, non-governmental organizations, and industry) is 
integrated into a total research program that complements the scientific niche filled 
by EPA’s own scientists. This provides a more nimble resource that is available to 
work on existing and emerging environmental issues than would be available with 
only an intramural or an extramural program. Changes in this budget, especially 
to STAR, will significantly impair the balance of this integrated research program 
in both core vs. problem driven and intramural vs. extramural research dimensions. 

c. Building Decontamination and Homeland Security 

Another surprising change for which no satisfying explanation was offered was 
EPA’s decrease in its commitment to Homeland Security. The Board believes that 
EPA must play a continuing role in Homeland Security in several areas. EPA’s 
building decontamination research is one of EPA’s contributions to Homeland Secu- 
rity and it is being eliminated in an $8.3 million dollar cut prior to its completion. 
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The Board is unsure as to why research on this important issue is being cut when 
it appears that further research is needed to provide rapid, on-target responses to 
contamination episodes. The Board is aware that other agencies have substantial re- 
sources devoted to Homeland Security, so perhaps other groups have taken on this 
role for the future. EPA has the special expertise to carry out this research. How- 
ever, if it is judged that this is not a research direction for EPA, it is still important 
to ensure that this work be conducted somewhere. 

2. Program Planning and Measurement 

a. Program Assessment 

Each year, the Board tries to evaluate EPA’s research priorities and their role in 
meeting the Agency’s goals. As part of the current review, the Board was given in- 
formation resulting from the application of a new survey tool, the Program Assess- 
ment Rating Tool (PART) that was used to evaluate selected EPA programs. The 
Board is concerned that decisions are being made about research program funding 
on the basis of the application of this new tool. 

To be clear, the Board did not receive or review information on the rating instru- 
ment itself; however, after evaluating PART summaries for several research pro- 
grams, our conclusion is that PART may, at this time, have a limited capacity to 
inform budget decisions on research programs. The Board’s is concerned with the 
manner in which the weighting formula in PART seems to influence the full anal- 
ysis and thus favor programs with short-run results over those having long-term re- 
sults. There is also concern that an evaluator’s subjective considerations might be 
able to bias those weights and the rating itself. 

Specifically, it appears that the weighting formula in the PART favors programs 
with near-term benefits at the expense of programs with long-term benefits. Since 
research inevitably involves more long-term benefits and fewer short-term benefits, 
PART ratings serve to bias the decision-making process against programs such as 
STAR ecosystem research, global climate change research, and other important sub- 
jects. The PART seems to be intended as a formula for predictions about likely pro- 
gram success. However, the weights that the PART assigns to different program 
characteristics do not seem to have been validated systematically against the con- 
tribution of each program characteristic to any independent objective measure of 
program success. If the weights in the tool are arbitrarily assigned, the PART may 
have characteristics that could lead to biases in evaluation that are related to the 
subjective judgments of its designers. We believe that the tool should be reviewed 
to determine its adequacy for its use in supporting budget decisions. 

As the Board observed significant decreases in science and research funding, it 
also noted a substantial resource increase in the State and Tribal Assistance Grant 
account (STAG) for an initiative for retrofitting school busses. The Board does not 
challenge the worthiness of this program, rather it notes that it has no information 
on the science supporting this initiative. The Board trusts that the benefits of this 
program have been rigorously reviewed. 

The real issue here is how research programs (and others) are to be evaluated 
and whether a different metric is necessary for basic vs. applied research programs. 
Also, of interest is whether research results should be evaluated separately from the 
outcomes of programs they are intended to support? Although the Board did not di- 
rectly evaluate the PART itself, it is of obvious difficulty to conceive of a simple 
quantitative metric that could be applied across the broad areas of ecosystem qual- 
ity, human health effects, endocrine effects, and technology development. The ques- 
tion is even more complex when you consider that some research is intended to de- 
velop limited data in the short-run to fill a specific knowledge gap and other re- 
search is intended to provide an understanding of whole systems in the long-term. 
Research program measurement is even more difficult because the knowledge and 
methods developed by EPA, especially ORD’s researchers, are not usually directly 
applied by ORD, rather they are often used by others to support decisions on a 
broad suite of diverse statutory mandates. Thus, we believe that evaluations of the 
performance of research programs will need to consider the specific factors of each 
program that the research is intended to support. Further, it is unlikely that simple 
formulas will be able to handle this task well. It is more likely that realistic re- 
search program performance assessment will need to be a combination of quan- 
titative metrics and other information and analyses which is then evaluated by 
groups of experts with relevant knowledge. 

I note that the NAS, in its review of STAR, also had concerns with quantitative 
routines used in performance assessments and noted that “The Committee judges 
that expert review by a group of people with appropriate expertise is the best meth- 
od of evaluating broad research programs, such as the STAR program.” 
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b. Multi-Year Plans (MYPs) 

Multi-Year Plans (MYPs) are an important innovation in EPA’s research planning 
process. The SAB has reviewed a limited number of these plans and the process 
used in their development and we believe that they will become more useful to the 
Board’s evaluation of EPA’s science and research, and its funding, in the future. 
MYP’s are tools that identify knowledge and methodology gaps needed to support 
EPA’s mission areas and the body of research that would address those needs. Fur- 
ther, they provide a basis for identifying annual performance goals and measures 
for efforts that become a part of EPA’s research budget. Finally, MYPs are very use- 
ful in providing focus on long-term progress toward research goals, especially on 
cross-cutting subjects such as pollution prevention where coordination across the 
Agency is essential. The Board supports the continued refinement of Multi-Year 
plans and is available to continue its review of EPA’s progress in this regard. 

3. EPA Science 

My final comment will address an issue of great concern to the Board. Our anal- 
yses of EPA’s science and research budgets, today and in the past, convince us that 
the Agency is in danger of underestimating the pace of large scale changes that are 
now occurring in our society. If so, EPA and the Nation are at risk of repeating the 
mistakes of the past that force us to spend huge sums of public funds to reduce and 
to clean up the pollution brought on by the first industrial revolution. 

The evidence before us suggests that we are now in a new, high velocity techno- 
logical revolution that will yield great economic gains, but at the same time, will 
offer new environmental challenges. Nanotechnology and biotechnology, to name 
only two innovations, are proceeding with breathtaking speed, and are compounded 
by forces such as global transfer of pollution and disease, and possible climate 
change. EPA must carefully examine all of its science and research programs and 
ask whether the Agency is conducting research that will help us protect human 
health and the environment while encouraging innovation and growth. 

This is not to say that EPA should neglect the “legacy” issues of the past; rather, 
the Agency must continue to resolve those problems, and at the same time, work 
with citizen’s groups, industry, and academia in creative ways to ensure that the 
Nation avoids a new legacy of human health and environmental problems. 

The Board would be pleased to work with EPA to explicitly address ways in which 
EPA science and research might be focused to help EPA develop, and use to the full- 
est, knowledge that will be instrumental in avoiding a negative legacy. 

I want to express my gratitude to the Members of the Subcommittee for inviting 
me to testify about EPA’s science and research. I would be pleased to answer your 
questions. 

Thank you. 


Biography for Genevieve Matanoski 

Dr. Matanoski is a Professor of Epidemiology at the Johns Hopkins University 
School of Hygiene and Public Health in Baltimore, MD. For a time after medical 
school she pursued a career in pediatrics and general preventive medicine. After 
earning a Doctor of Public Health Degree, she was appointed to the faculty of Johns 
Hopkins University and has been a professor since 1976. In addition to teaching and 
research, Dr. Matanoski has had appointments in a number of teaching and train- 
ing programs in the U.S. and abroad and is a frequent advisor to legislative and 
policy-making groups. She is a member of several scientific advisory bodies both for 
governmental agencies and for industry. She is a past Chair of the EPA Science Ad- 
visory Board, as well as a past Chair of the SAB Radiation Advisory Committee. 
She now serves as a member of the Board of the EPA SAB. During her tenure on 
the EPA SAB, Dr. Matanoski has been involved in the writing of several documents 
produced to provide advice to EPA including the “Beyond the Horizon: Using Fore- 
sight to Protect the Environmental Future” document and the Integrated Risk 
Project report “Toward Integrated Environmental Decision-making,” and was Chair 
of the latter Committee. She is the author or co-author of over 80 publications. 

Dr. Matanoski’s work has focused on the epidemiology of cancer, including blad- 
der, lung, skin and uterine cancers, and leukemia. Her research studies have exam- 
ined the risks associated with occupational and environmental exposures to such 
agents as radiation, electromagnetic fields, and chemical substances as styrene, bu- 
tadiene, arsenic and environmental tobacco smoke. Her recent research has empha- 
sized reproductive effects and congenital malformations from environmental expo- 
sures. Her early work involved infectious diseases and illnesses in infants and chil- 
dren. Dr. Matanoski received a B.A. degree in chemistry at Radcliffe College and 
a M.D. at the Johns Hopkins School of Medicine. She also earned a Doctor of Public 
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Health Degree from the Johns Hopkins University School of Hygiene and Public 
Health (11/2003). 

Chairman Ehlers. Thank you for your testimony. 

Dr. Denson. 

STATEMENT OF DR. COSTEL D. DENSON, PROFESSOR, DE- 
PARTMENT OF ENGINEERING, UNIVERSITY OF DELAWARE; 

MEMBER OF THE NATIONAL RESEARCH COUNCIL PANEL 

REVIEW, THE MEASURE OF STAR: REVIEW OF THE EPA’S 

SCIENCE TO ACHIEVE RESULTS (STAR) RESEARCH GRANTS 

PROGRAM 

Dr. Denson. Good morning, Mr. Chairman and Members of the 
Committee. 

My name is Costel Denson. I am a Professor of Chemical Engi- 
neering at the University of Delaware, where I have been employed 
since 1977. I have also served as Vice Provost for Research. In that 
position, I had oversight of and was responsible for all aspects of 
the research enterprise at the University. Recently, I served as a 
member of the National Research Council, the NRC Committee, to 
review EPA’s Research Grants Program. The NRC is the operating 
arm of the National Academies of Sciences and Engineering. I am 
pleased to be here to discuss the unanimous findings and rec- 
ommendations of that committee. 

The Environmental Protection Agency is a mission agency estab- 
lished to protect human health and to safeguard the natural envi- 
ronment. EPA’s regulatory and decision-making role requires that 
the agency have access to the best available science that is relevant 
to its mission. In an effort to improve the scientific foundation of 
its decision-making process, the agency’s Office of Research and 
Development established the Science to Achieve Results program, 
the STAR program, as part of its Research Grants Program in 
1995. 

STAR is a competitive, peer-reviewed, extramural research 
grants program created to encourage interagency collaboration and 
to increase EPA’s access to the Nation’s best scientists and engi- 
neers in academic and other non-profit research institutions. It 
supports research pertaining to human health and the environment 
and it is designed to maximize the independence of the researchers 
it supports and to provide an equal opportunity for all researchers 
to qualify for that support. 

Over the past six years, a number of occasions have arisen where 
I have had the opportunity to review the STAR program. When I 
was chair of EPA’s Office of Research and Development’s Board of 
Scientific Counselors, I oversaw the review that BOSC conducted 
in 1997 of EPA’s National Center for Environmental Research and 
Quality Assurance, and the STAR program, which was operated by 
that center. Again, in 1999/2000, while still chair of BOSC, I 
oversaw a review of the STAR program that BOSC and EPA’s 
Science Advisory Board conducted jointly. Now as I mentioned ear- 
lier, I served on the NRC Committee to Review EPA’s Research 
Grants Program. The findings and conclusions from this most re- 
cent report, along with those from the previous BOSC reports, lead 
me to conclude the following. 
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First, environmental regulatory decisions must be informed by 
the best science. The STAR program is judged to be the best mech- 
anism that we have for providing the very best science through ex- 
tramural sources. Second, research in STAR is focused on EPA’s 
and the country’s greatest environmental needs. And finally, the 
STAR program has an exceptional process for the peer-review of 
proposals. The NRC Committee stated in its conclusions that the 
STAR program “compares favorably with and in some cases ex- 
ceeds that in place at other agencies that have extramural research 
programs, such as the National Science Foundation and the Na- 
tional Institute of Environmental Sciences Health,” which is part 
of NIH. 

The Committee, in its deliberations, developed its own metrics, 
and in that case, developed a series of nine questions, and for each 
question, there was a finding and a recommendation. Those are de- 
lineated in my written testimony, and also as a part of this report, 
which is called The Measure of STAR: Review of the U.S. Environ- 
mental Protection Agency’s Science to Achieve Results Research 
Grants Program. 

Mr. Chairman, that concludes my testimony. I would be pleased 
to answer any questions. Thank you and the Members of your 
Committee for the opportunity to participate in this hearing. 

[The prepared statement of Dr. Denson follows:] 

Prepared Statement of Costel D. Denson 

Good morning, Mr. Chairman and Members of the Committee. My name is Costel 
Denson. I am a Professor of Chemical Engineering at the University of Delaware 
where I have been employed since 1977. I have also served there as Vice Provost 
for Research. In that position I had oversight of, and was responsible for, all aspects 
of the research enterprise at the university. Recently, I served as a member of the 
National Research Council (NRC) Committee to Review EPA’s Research Grants Pro- 
gram. The NRC is the operating arm of the National Academies of Sciences and En- 
gineering. I am pleased to be here to discuss the unanimous findings and rec- 
ommendations of that committee. 

The Environmental Protection Agency (EPA) is a mission agency established to 
protect human health and to safeguard the natural environment. EPA’s regulatory 
and decision-making role requires that the agency have access to the best available 
science that is relevant to its mission. In an effort to improve the scientific founda- 
tion of its decision-making process, the agency’s Office of Research and Development 
established the Science to Achieve Results (STAR) research grants program in 1995. 

STAR is a competitive, peer-reviewed, extramural research grants program cre- 
ated to encourage interagency collaboration and to increase EPA’s access to the Na- 
tion’s best scientists and engineers in academic and other nonprofit research institu- 
tions. It supports research pertaining to human health and the environment and is 
designed to maximize the independence of the researchers it supports and to provide 
an equal opportunity for all researchers to qualify for support. 

Over the past six years a number of occasions have arisen where I have had the 
opportunity to review the STAR Program. When I was chair of EPA’s Office of Re- 
search and Development’s Board of Scientific Counselors (BOSC), I oversaw the re- 
view that BOSC conducted in 1997 of EPA’s National Center for Environmental Re- 
search and Quality Assurance, and the STAR Program, which was operated by that 
center. Again, in 1999/2000, while still chair of BOSC, I oversaw a review of the 
STAR program that BOSC and EPA’s Science Advisory Board conducted jointly. And 
now, as mentioned earlier, I served on the NRC Committee to Review EPAs Re- 
search Grants Program. 

The findings and conclusions from this most recent report, along with those from 
the previous BOSC reports, lead me to conclude the following: 

1. Environmental regulatory decisions must be informed by the best science: 
the STAR program is judged to be the best mechanism that we have for pro- 
viding the very best science through extramural sources. 
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2. Research in STAR is focused on EPA’s and the country’s greatest environ- 
mental needs. 

3. The STAR Program has an exceptional process for the peer-review of pro- 
posals. The NRC committee stated in its conclusions that the STAR program 
“compares favorably with and in some cases exceeds that in place at other 
agencies that have extramural research programs, such as NSF and NIEHS 
(NIH).” 

In 2000, EPA asked the NRC to conduct an independent assessment of the STAR 
program. In response, the NRC established a committee and gave it the following 
tasks: 

Assess the program’s scientific merit, its demonstrated or potential impact on 
the agency’s policies and decisions, and other program benefits that are relevant 
to EPA’s mission; 

Recommend ways to enhance the program’s scientific merit, impact, and other 
benefits; and 

In the context of other relevant research conducted or funded by EPA, and in 
comparison with other basic and applied research grants programs, address the 
STAR program’s research priorities, research solicitations, peer-review process, 
ongoing research projects, and results and dissemination of completed research. 

In 2003, the committee completed its report, The Measure of STAR, Review of the 
U.S. Environmental Protection Agency’s Science to Achieve Results (STAR) Research 
Grants Program. 

In its evaluation, the committee developed a series of nine specific questions that 
it considered were of greatest importance to the research program. I will present 
each of these questions, along with the committee’s overall findings and rec- 
ommendations for the STAR program. 

1. Should the STAR program continue to be part of EPA’s research pro- 
gram? 

Finding. EPA requires a strong and balanced science and technology research pro- 
gram to fulfill its mission properly. The STAR program is an important part of the 
overall EPA research program. 

The STAR program is EPA’s pre-eminent program that solicits independent sci- 
entific and technologic research from the Nation’s best academic and other nonprofit 
research institutions. The program has established and maintains a high degree of 
scientific excellence. Through broadly advertised, competitively awarded, peer-re- 
viewed grants, the STAR program provides the agency access to independent infor- 
mation, analyses, and perspectives. 

The STAR program provides the agency access to a broad community of research- 
ers, allows it to fund research at the cutting edge of science, and assists it in ad- 
dressing information gaps that it does not have the internal resources to address 
properly. The STAR program also encourages its grantees to disseminate their re- 
search results widely in peer-reviewed scientific journals. 

Recommendation. The STAR program should continue to be an important part of 
EPA’s research program. 

2. What is the unique contribution of the STAR program? 

Finding. The STAR program funds important research that is not conducted or 
funded by other agencies. 

For instance, the STAR ecologic-indicators program is the primary source of sup- 
port of research on the development of water-quality indicators for biologic moni- 
toring. The interdisciplinary centers that STAR has funded also represent an inno- 
vative approach to supporting research that will be most relevant for environmental 
decision-making. 

The STAR program has also made commendable efforts to leverage funds by es- 
tablishing research partnerships with other agencies and organizations that have 
similar or complementary research interests, including the National Science Foun- 
dation, the Department of Energy, the Office of Naval Research, the National Aero- 
nautics and Space Administration, the National Oceanic and Atmospheric Adminis- 
tration, the U.S. Department of Agriculture, the National Institute of Environ- 
mental Health Sciences, the Department of Interior, and the American Waterworks 
Research Foundation. 

Recommendation. STAR should continue to partner with other government and 
non-government organizations to support research of mutual interest and of rel- 
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evance to EPA’s mission, explore innovative approaches for carrying out this re- 
search, and sponsor a diverse portfolio of research that alerts the agency to emerg- 
ing issues and provides independent analyses of issues that the agency is currently 
addressing. 

3. Does the STAR program have adequate processes to ensure that it is 
sponsoring high-quality and relevant research? 

Finding. The STAR program has developed a grant-award process that compares 
favorably with and in some ways exceeds that in place at other agencies that have 
extramural research programs, such as the National Science Foundation (NSF) and 
the National Institute of Environmental Health Sciences. An unusually high degree 
of planning goes into identifying the specific research subjects to be supported. The 
agency also puts considerable time and thought into preparing effective research so- 
licitations and into funding projects that are relevant to its mission and program 
needs. Furthermore, the STAR program has established a rigorous peer-review proc- 
ess. 

Recommendation. The STAR program should continue to improve the focus of its 
Request for Applications, and when the agency does not have the capacity internally 
to adequately define the state of the science in a particular research field, STAR 
should consider greater use of external experts to assist in identifying the highest- 
priority research and data gaps. 

4. Is the STAR program producing high-quality research results? 

Finding. Although it is still too early for comprehensive evaluations of the research 
results of the STAR program, some STAR research efforts have already substan- 
tially improved the scientific foundation for decision-making. 

Many STAR projects have resulted in articles in highly respected, peer-reviewed 
journals a traditional measure of research quality. These STAR research results 
have already helped to improve our understanding of the causes, exposures, and ef- 
fects of environmental pollution information critical to improving the scientific foun- 
dation for decision-making. For instance, STAR-funded research on particulate mat- 
ter has helped to improve our understanding of the biologic mechanisms by which 
inhaled ambient particles cause health effects and the nature of some of these ef- 
fects. These data are critical to future regulatory decisions regarding our nation’s 
ambient air quality. 

Recommendation. EPA should continue its efforts to attract “the best and the 
brightest” researchers to compete for STAR funding. 

5. Are the STAR program results useful for EPA decisions and processes? 

Finding. The STAR portfolio effectively supports EPA’s mission and research and 
development strategic plans and GPRA goals. Specific STAR research projects have 
yielded significant new findings and knowledge critical for regulatory decision-mak- 
ing. 

The STAR program is too young to be able to document fully the extent to which 
its research results are being used to inform development of new regulations and 
environmental-management decisions. However, some STAR projects have already 
yielded information important for environmental decision-making. For example, 
STAR-sponsored research in endocrine disruptors, particulate matter, and ecologic 
assessment has resulted in groups of peer-reviewed publications of immediate use 
in understanding causes, exposures, and effects of environmental pollution. Those 
results are directly relevant to EPA’s mission to “protect human health and to safe- 
guard the natural environment — air, water, and land — upon which life depends.” 
For instance, research on ecologic indicators has led to the development of a dy- 
namic, economically linked model to evaluate the driving forces and ecologic con- 
sequences of land-use change. 

Recommendation. The STAR program and EPA’s Office of Research and Develop- 
ment should develop mechanisms for documenting the extent to which its research 
is being used to support the agency’s environmental decision-making, should con- 
sider using outside experts to help document systematically the “state of the 
science” before research is initiated, and should synthesize the results of the re- 
search when it is completed to identify the specific contributions that STAR and 
other EPA research has made to providing critical information. 
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6. Is the STAR program effective in providing results relevant to the ap- 
propriate audiences? 

Finding. The STAR program has been commendably aggressive in experimenting 
with innovative approaches to communicating the results of its funded research to 
a wide variety of users and audiences, but its success in these efforts has been un- 
even. 

The STAR program supports research of potential value to a variety of users and 
audiences, both in and outside EPA. Much of the research is aimed primarily at the 
scientific community and those responsible for providing technical support for envi- 
ronmental-management decisions. 

The program, however, also has other potential users, at least for some of its re- 
search results, including other federal agencies; industry; State, tribal, and local 
governments; nonprofit environmental organizations; and international environ- 
mental agencies. 

The STAR program has experimented aggressively with a wide variety of commu- 
nication mechanisms, including EPA’s web site and publication of reports. Through 
these efforts, the program appears to substantially exceed the dissemination efforts 
of most other research-sponsoring organizations, both in and outside the Federal 
Government. Nevertheless, the STAR program could substantially improve its dis- 
semination efforts by directing them more effectively to specific users and audiences. 

Recommendation. The STAR program should clearly identify the intended audi- 
ences for proposed research results as early in the process as possible and indicate 
them in the Request for Applications. When appropriate, EPA should consider in- 
volving representatives of the intended audiences from outside the agency to help 
define the relevant research results and a strategy for their dissemination. 

7. Should the fellowship program continue to be part of EPA’s research 
program? 

Finding. The STAR fellowship program is a valuable mechanism for enabling a 
continuing supply of graduate students in environmental sciences and engineering 
to help build a stronger scientific foundation for the Nation’s environmental re- 
search and management efforts. 

It is the only federal fellowship program exclusively designed for students pur- 
suing advanced degrees in environmental sciences. 

Recommendation. Given the Nation’s continuing need for highly qualified sci- 
entists and engineers in environmental research and management, the STAR fellow- 
ship program should be continued and funded. 

8. Are the STAR program’s funds adequate to achieve its objectives? 

Finding. STAR is only able to fund less than 15 percent of the proposals received 
for its individual investigator and center grants, and its funding has not kept pace 
with the rate of inflation. 

NIH and NSF strive to fund, on the average, 25-30 percent of the proposals re- 
ceived. STAR’S budget allows it to fund only 10-15 percent of the proposals it re- 
ceives and only about 60 percent of those rated “excellent” or “very good” by its inde- 
pendent quality peer-review panels. By that measure, STAR does not have sufficient 
funds to recognize all the best proposals received. 

Although the STAR program’s budget grew rapidly in its first three years, it has 
not kept pace with general inflation in the last few years. That is particularly true 
of the STAR fellowship program. The effect of that budgetary situation is exacer- 
bated by the fact that costs of research have outpaced general inflation for more 
than a decade. Therefore, at present, STAR funds buy less research than the same 
amount of money could have bought several years ago. 

It is appropriate to consider the funding of the STAR program in the context of 
the overall funding for all of EPA’s Office of Research and Development, which also 
has not kept pace with inflation. STAR currently represents about 18% of EPA’s Of- 
fice of Research and Development total funding. The committee considers that per- 
centage to be a reasonable recognition of the value of independent peer-reviewed re- 
search to the agency. 

Recommendation. STAR program funding should be maintained at 15-20 percent 
of the overall research and development budget, even in budget-constrained times. 
However, budget planners should clearly recognize the constraints of not having in- 
flation escalators to maintain the level of effort of the entire program. 
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9. How should the STAR program be evaluated? 

Finding. There are no easy answers when it comes to identifying metrics for evalu- 
ating research programs, and the best approach for evaluating the STAR program 
is to establish a structured system of reviews by panels of experts. 

The committee assessed the quality, relevance, and performance of the STAR pro- 
gram, as set forth in recent OMB research and development criteria, by using quali- 
tative and quantitative metrics. That is one approach for reviewing the STAR pro- 
gram and similar programs. Several examples of qualitative and quantitative 
metrics that were used for evaluating the STAR program are: Does the STAR pro- 
gram have a clearly defined plan for regular, external reviews of its research qual- 
ity, and has this plan been effectively carried out? Has the program made signifi- 
cant contributions to advancing the state of the science in particular research top- 
ics? Does the program award grants expeditiously? Does the program have a sched- 
ule for the products it intends to produce and how well is it adhering to the sched- 
ule? 

The committee’s judgment is that quantitative metrics, although outwardly sim- 
pler to use, are not necessarily more informative than qualitative metrics. In some 
cases, quantitative metrics can be misleading, and emphasizing inappropriate 
metrics can distort the research outputs of a program. Qualitative metrics are less 
likely to have such effects, but they need to be interpreted carefully. 

The committee judges that expert review by a group of people with appropriate 
expertise is the best method of evaluating broad research programs, such as the 
STAR program. Expert review is appropriate for evaluating both the processes and 
the products of the STAR program. Both qualitative and quantitative metrics can 
provide valuable support for such expert reviews. 

Recommendation. STAR and EPA’s Office of Research and Development should 
establish a structured program of reviews by panels of independent experts and 
should collect the appropriate information to support these reviews. 

Mr. Chairman, that concludes my testimony. I would be pleased to answer any 
questions. Thank you and the Members of your committee for the opportunity to 
participate in this hearing. 
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Discussion 

Chairman Ehlers. Thank you very much, and thank you to all 
of the witnesses for being here and for your testimony. 

We will now begin with the questions. The Chair will recognize 
himself first for five minutes, and then each Member, in turn, will 
receive a five-minute question period. If there are sufficient ques- 
tions, we will continue with a second round, as well. 

Rationale for Science to Achieve Results (STAR) 
Research Grants Program Budget Cuts 

The first question is for Mr. Johnson and Dr. Gilman. You 
have — been interesting in my career in the Congress. Because I am 
a scientist, Members are constantly coming to me to ask my opin- 
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ion on scientific issues and questions, and probably the most often 
repeated phrase that I have heard is I want some good science on 
this. Frequently, they will also say I want sound science, in which 
case I say I don’t know anything about acoustics. But I do know 
something about good science, and I have been actively involved in 
trying to improve the scientific effort of the Nation in every agency, 
including the EPA, and I am very pleased the EPA has made con- 
siderable progress. 

It is very disappointing to see the recommendations for the big 
cuts that we are seeing recommended here. The 92 percent I re- 
ferred to earlier and the 35 percent in STAR, and the question for 
Mr. Johnson and Dr. Gilman is why the cuts? Where does this 
come from? This is not just a reallocation. This is really a cut, un- 
less you can show me that you are transferring money to some 
other agency to do the same type of research. Particularly in the 
environment, Members of Congress are very interested in good 
science. They want it done right. They want to know the right an- 
swer so they can make the right decision, so I would appreciate 
comments from both of you on this issue. 

Mr. Johnson. The more substantive answer will come from Dr. 
Gilman, I suspect, but let me just give you my general view of this. 

As we mentioned, we think the focus should be on what we are 
getting, what the result is, what we are learning and what — how 
we are advancing our usual pursuit of a cleaner, healthier environ- 
ment. The feeling was that the 20-some odd million dollars would 
be better spent in other programs than in the Ecological Research 
and Pollution Prevention Programs because we believe that there 
are more specific deliverables that could be resulting from those 
programs. We think those programs are well-funded now, and we 
believe that there would be little detriment to the output of those 
research programs with those cuts. We believe we could produce a 
greater return for the 20-some odd million dollars if we put those 
monies in the Prevention of Pesticides and Toxics Program. 

I think this President has demonstrated his commitment to all 
the things that EPA stands for and all of these environmental ob- 
jectives that we all share. This is — should not be considered to be 
a back-off in the President’s or the Administration’s commitment to 
a healthy, clean environment. It should be considered to be a focus 
on what we should be — what we are trying to get for our money, 
not how much money we are trying to spend. Now for the real an- 
swer, Dr. Gilman. 

Dr. Gilman. Well, there is no question there is competition for 
resources. The Agency is putting forward a number of aggressive 
efforts proposing a $150 million increase in Superfund, proposing 
a School Bus Diesel Retro-Fit Program on the order of $60, $65 
million. There is also an initiative in leaking underground storage 
tanks. You know, the list goes on. These are the kinds of things 
that create a competition for funds within the Agency. So in terms 
of where do we place the dollars, that is certainly an element for 
consideration. For the ecological research, the program still has a 
substantial effort beyond the cuts, as is the case in pollution pre- 
vention. So we think we can still drive those programs to have good 
work done and good outcomes from that work, even with these re- 
ductions. 
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Chairman Ehlers. But I am really puzzled because science is 
the basis for what you do, and it seems to me you are transferring 
money out of scientific research and into programs, and I just can’t 
believe that the science has been done. That you don’t have to do 
it anymore, and that it should be cut by that. I am particularly 
concerned about the STAR program, which has received high rat- 
ings. You heard two panelists here today make good comments 
about it, good reviews, and it is generally acknowledged as being 
one of the strongest environmental programs. Why cut STAR by 35 
percent? And I know there is always a tendency of managers to say 
well, we have got to protect our own employees, and therefore, we 
will cut the outside. But you get more for your money with STAR 
than I think you get almost anywhere else. 

What is the justification for cutting STAR? 

Dr. Gilman. Again, the decision wasn’t made on the basis that 
STAR is a bad program. Decisions were made on the basis of the 
program reviews and the elements of the ecological and pollution 
prevention programs. 

Chairman Ehlers. What are these program reviews you are re- 
ferring to? 

Dr. Gilman. The program reviews that we did in collaboration 
with the Office of Management and Budget’s PART exercise. 

Chairman Ehlers. So you are saying the cuts are a result of ap- 
plying PART? 

Dr. Gilman. Yes, for these two programs, that is the case, but 
not as it relates to performance of the STAR grant program per se. 
The reductions were made in the STAR grant programs because, 
just as those programs are the place where we can easily ramp up, 
it is also the place where, in terms of managing the entire research 
enterprise for the agency, we look to ramp down when we have to 
make a reduction. It is the place where we do the least disruption 
to our infrastructure for research when we have to make those re- 
ductions. 

Chairman Ehlers. But that implies that you are doing it be- 
cause it is easier to ramp down, even though some of the best 
science is done there. 

Dr. Gilman. Well, I wouldn’t say it necessarily is better than, 
though it sometimes is different than, the work that is going on in- 
side the Agency. But the alternative of cutting back on the re- 
sources for the Intramural Program or reducing the Intramural 
Program is for the longer run. When we are very hopeful of re-engi- 
neering these programs and getting them their better measures 
and their better performance — to begin to reduce the internal infra- 
structure when we hope to be able to turn these things around 
doesn’t make sense in our opinion. 

Chairman Ehlers. Well, it doesn’t make sense to me when the 
President is trying to freeze the discretionary non-defense, non-se- 
curity budget that we are cutting a major effort on your part. I 
mean, all of EPA is being cut, and that also doesn’t make sense to 
me. If we are freezing, let us freeze everything where it is, but let 
us not cut to this extent. 

My time has expired. Let me just add one point, and after spend- 
ing a fair amount of my life in science and part of it in managing 
science, I can assure you that managing science is one of the most 
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difficult enterprises around because it is very difficult to develop 
measures for measuring science because you really literally do not 
know what the outcome is going to be. And it takes a special type 
of expertise and I am very, very suspicious of instruments and 
methodologies that are applied across the board by any agency, 
whether OMB or something else, that just says okay, fill in the 
blanks and then we will make a decision. That does not do justice 
to the enterprise, and I think we have to be very cautious about 
that. 

Mr. Johnson, did you want to respond before I go on? 

Mr. Johnson. Yes, I would. I think we recognize that R&D needs 
to be looked at differently than an operating program, and so the 
questions are different. I have gotten to know Eli Zerhouni in the 
last year plus, and he talks about the opportunity to better manage 
our investments in research better than we are now, and he under- 
stands way better than I and comparable to your understanding as 
to how difficult it is to measure this, but these can be managed. 
Shame on us if we don’t continually try to deal with all of its dif- 
ficulty, try to get a better understanding of what we are getting 
and are we on track, and is there accountability? And we under- 
stand that the end product, especially of basic research, is by defi- 
nition an open question. But like every dollar we spend, we must 
continue to try to find better ways to make sure that the money 
is well spent. 

Chairman Ehlers. I don’t agree with your goal. I disagree with 
the result you have come up with. I have — my time has expired. 
I would like to — who came first? All right. Zoe, you were first. 
Please recognize Congresswoman Lofgren from California. 

Incorporation of STAR Scientific Research into 
Decision-making 

Ms. Lofgren. Thank you, Mr. Chairman. I thank you for holding 
this hearing today. I think it is an important one, and although the 
testimony has been a little bit dense, I think we can really distill 
what is going on here so that it is easily understood. 

I remember years ago a Member of the House giving me a phrase 
about OMB. That they know the cost of everything and the value 
of nothing, and I think we have got that kind of fight brewing 
here — I don’t mean disrespect, but it is a word we use often in the 
House, the bean-counters versus the scientists, and when it comes 
down to good science, I am going to rely more on the scientists than 
I am on the CPAs. And it does disturb me that the basic research 
that we are — have gotten through the STAR program, which has 
been evaluated by the scientists, and Dr. Denson, I found your tes- 
timony, written testimony particularly helpful because as you went 
through the analysis that has been undertaken of the program, it 
is not without rigor, and it — but it is not mechanistic. It has been 
an analysis that actually is thoughtful. It is not, you know, check 
off the boxes, and I think that is the kind of analysis that we need 
to have. 

And that doesn’t mean that changes won’t sometimes result be- 
cause I know — I am not a scientist, as the Chairman is, but I have 
certainly — I come from Silicone Valley. I live surrounded by sci- 
entists, and there is not a more vicious critic of science than an- 
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other scientist. So that can certainly result. But I am just sort of 
wondering, when you look at these nine specific results, on page 
eight of your testimony, really the — one of the things that you dis- 
covered was not that the science that was being developed was 
flawed or defective, but there were big questions about the extent 
to which the research was actually being used, I am quoting you, 
“to support the agency’s environmental decision-making.” So the 
question, as I understood it, was really not about the science deliv- 
ered through the STAR program, but the agency’s inability or ques- 
tioning their ability to actually incorporate the data in their deci- 
sion-making. 

Is that a fair analysis of that testimony? I don’t want to misquote 
you, but that is the way it seemed to me. 

Dr. Denson. I think that is an accurate assessment. I would add 
one other thing, and that is that it takes a while for the research 
results to percolate through the system and to apply the results to 
some known situation. These results, the research results in many 
ways are autocatalytic in that they feed on each other, and so once 
you have a set of results, a scientist someplace reads those results 
and has another idea that yet takes it to another level. And finally, 
it gets to a critical mass where it becomes very obvious how these 
results can be used to implement a particular program. 

My own sense, and this is my personal reflection since I have 
been involved with it since nearly the start, is that the STAR and 
the agency have made great strides in improving the program and 
improving the way it implements the results. They have a way to 
go yet. I think the STAR program has some improvements to make, 
and we have mentioned what those are. But in every step along the 
way that I have been involved in the evaluation of STAR, when 
those comments have been made, the agency has responded very 
promptly and forthrightly in trying to make the changes and the 
improvements. 

Effect of Ecological Research Programs Budget 
Cuts on Human Health Research Programs 

Ms. Lofgren. Well, listening to your testimony and really the 
testimony of all the witnesses, it seems that the extramural por- 
tions of EPA’s research efforts, including the research on ecological 
systems, I am particularly interested in the mercury analysis, are 
basically just not going to happen if the STAR program sustains 
the kind of cuts that are recommended. You are a prominent sci- 
entist, and I thank you very much for volunteering to do this for 
so many years. Is that your assessment as well? 

Dr. Denson. I believe that — I believe personally that it would be 
detrimental to cut the ecological programs for several reasons. One 
is that you have to recognize that EPA and ORD is a regulatory 
agency, and conducting research in a regulatory agency is different 
than conducting research in other kinds of agencies. One feeds on 
the other. It is a coupling effect. What happens in research on 
human health informs what happens in research and the results on 
the ecology, and vice-versa. So they are intimately coupled. And if 
you cut one, then in an agency of this sort, I think you seriously 
undercut the ability to do research across the board. 
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My second point is this, is that the National Science Foundation 
conducts, in the environmental area, research on ecological prob- 
lems. I mean, they don’t do human health research. 

Ms. Lofgren. Right. 

Dr. Denson. The National Institute of Health does human health 
research. So we have two compartments. And what EPA, ORD is 
doing is to bring those two together because they are in a regu- 
latory agency and they have to conduct research, one arm which 
informs the other. So to cut the ecological part, impairs EPA’s and 
ORD’s ability to carry out its mission. See, because that is part of 
their mission, and they need that, as we said in this report, they 
need a strong research arm. 

Ms. Lofgren. I know my time is over, Mr. Chairman, but I 
would just note that at a time when there has been a wide concern 
about proposals to increase the amount of mercury that can be per- 
mitted in the environment, I think the American people are going 
to be very suspicious of a proposal to cut research into the mercury 
element, while, you know, polluters are being promised relief for 
more mercury in the environment. I think it is a terrible message 
to give to the American people who I don’t think ever asked for 
more pollution in their children. 

Chairman Ehlers. The gentlewoman’s time has expired. Pleased 
to recognize the gentleman from Texas, Mr. Burgess. 

Immediate Effects of Science to Achieve Results 
(STAR) Grants Program Budget Cuts on EPA Re- 
search 

Mr. Burgess. Thank you, Mr. Chairman, and I also thank you 
for calling this hearing today. 

Let me first ask Dr. Gilman if the Science to Achieve Results, if 
the cuts in that program are implemented as written, what do we 
lose? What research are we going to — what is going to be the im- 
mediate effect of that? 

Dr. Gilman. There are a set of requests for proposals that we 
would have otherwise put out on the street that we will not put out 
on the street, but none of the cuts that we are proposing to the Ex- 
tramural Program would, in and of themselves, terminate research 
overall. So there is still a substantial ecological research effort un- 
derway. It behooves us to do more towards collaborating with some 
of our other federal agencies to stretch the dollars we have there. 
In the case of pollution prevention, we have a substantial Pollution 
Prevention Program remaining. We are in the process of re-engi- 
neering that program, and there is a true connection between the 
results of the PART process and our efforts to re-engineer that 
process and change its focus. So none of the cuts to the program 
will terminate initiatives in either ecological research or in the pol- 
lution prevention research. 

Mr. Burgess. Well, then taking the last point from the 
gentlelady from California, will the mercury research and the re- 
search on human endocrine disruptors, will that continue? 

Dr. Gilman. That research does continue. The mercury research 
is a good example. Actually, we received a cut from the Congress 
in our proposed budget for 2004, which made us cut back on an ini- 
tiative we had proposed in mercury-related research. In effect, the 
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proposal for 2005 would take that off the table, as the Congress did 
as well, but we still have an intramural program. The big focus of 
our intramural program at this point is trying to better understand 
how we can do continuous monitoring of emissions for mercury so 
that we can put in place the regulatory program that has been pro- 
posed. The regulatory program that has been proposed for mercury 
reduction does constitute the largest reduction that has ever been 
proposed, and it is the first time that power companies will be re- 
quired to reduce their mercury emissions. 

Mr. Burgess. Okay. Let me be clear on that then, that program 
is going to continue to go forward. Is that correct? 

Dr. Gilman. Yes. 

Mr. Burgess. All right. Then just to give us a flavor of what is 
on the shelf in those requests for proposals, what are some of the 
things that will not be brought forth as a result of the reduction 
in funds? 

Dr. Gilman. In substance, there is no one particular body of re- 
search that won’t go forward. It is the supplemental work that 
would be done by extramural researchers. There is no question that 
that supplemental research would be of value to us, but it is not 
anything that we can’t make up for in the future, in my opinion. 

Mr. Burgess. Very well. Dr. Denson, and too, I thank you for 
providing such a comprehensive but readable paper for us. You al- 
lude in here to that you do research that is not funded by other 
agencies, but there is also some discussion about leveraging your 
funding and partnering either with I guess other agencies or with 
private, nongovernmental organizations. Can you talk to us about 
that just a little bit? If the proposed cuts go through, how are you 
prepared to carry forward to make sure to maximize your research 
dollars and to leverage those dollars so that they are spent effec- 
tively? 

Dr. Denson. I would speak to that question, sir, as a Member 
of the NRC Committee. If I understand the question correctly, my 
response would be this. That one of the areas where we gave STAR 
program very high marks was in the area of partnerships and col- 
laboration. The partnerships allowed STAR program to greatly le- 
verage the funds that they have, and in particular, in terms of the 
ecology, I am thinking of leveraging the funds to the National 
Science Foundation and the Program for Technology for Sustain- 
able Environment. So areas where there are not a great deal of re- 
sources in the STAR program, they have been very diligent with 
the money they do have in leveraging it with other organizations. 
As I said, NSF is a good example because they only work on eco- 
logical problems, on environmental problems. 

And the STAR program has been diligent enough to take what 
resources they have to leverage those with NSF. If those monies 
are cut, then one would have to look at the leveraging effect be- 
cause not only do you lose the money that is cut, you lose what you 
could have gotten in that leveraging effect. 

Mr. Burgess. Very well. I see my time has expired. I take your 
point that we will lose the multiplier effect. I want to thank the 
panel for doing this today, being so well-prepared. It has been very 
informative to me, and I will yield back. 
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Application of the Program Assessment Rating Tool 
(PART) and Alternative Review Methods Within 
EPA’s Office of Research and Development 

Chairman Ehlers. The gentleman’s time has expired. Mr. Davis 
has stepped out of the room temporarily, so I will proceed with my 
next questions and we will recognize him, should he return. 

I just wanted to note, Dr. Matanoski, you in your testimony said 
that PART appears to favor short-term R&D programs with near- 
term benefits at the expense of programs of long-term benefits. I 
also noticed Mr. Posner has also had some criticisms of PART, and 
the real question is what is a good evaluation program? Mr. Gil- 
man, you said that EPA is committed to developing good standards 
by which to measure EPA’s R&D programs. A question, Mr. John- 
son, is what standards would OMB find acceptable? It seems to be 
the thinking of the scientists here and GAO that PART is not ap- 
plicable as it has been used to the scientific programs. I am won- 
dering, and I am sympathetic to good management of science. I al- 
ways have been. 

The question is can we do it right? I remember we worked very 
hard in this Committee when GPRA came out to modify GPRA as 
it is applied to science so that it made sense, and after some stum- 
bles, I think that has begun to work. The question is, Mr. Johnson, 
what standards would OMB find acceptable if we are going to im- 
prove this? How can we make PART fit science, rather than the 
present situation where we have at least expressed opinions here 
that PART doesn’t really fit in this situation? The Floor is yours. 

Mr. Johnson. I can’t speak, maybe Dr. Gilman can, on what spe- 
cific opportunities we have to improve the R&D criteria that are 
addressed in the PART R&D, R&D PART. But I do know Dr. 
Zerhouni is supportive of use of the PART in assessing various re- 
search programs, and the PART will get better and better each 
time. There is nothing magic about the PART, these 25 questions. 
We think these are good questions now, and they will get better 
every year as we get smarter and smarter, and the goal here is to 
look at, on a regular basis, everything we spend money on, includ- 
ing science, and ask ourselves is this consistent with our priorities? 
Do we think we are getting something for our money? Is the re- 
search money of these long-term projects being well-managed, and 
if not, what can we do about it? 

But I don’t know the answer to your specific question. 

Chairman Ehlers. Okay. I will make a comment, but let me let 
the others comment first, and we will go through the whole panel. 
Dr. Gillman? 

Dr. Gilman. Okay. Mr. Chairman, EPA is not GPRA-phobic. We 
are not afraid to be measured. We, the science arm have tried 
hard. We need to do better. One of the things we have done is a 
pilot with the Office of the Inspector General, trying to develop a 
way of planning our program and measuring our program that 
really does look to long-term outcomes. That is a pilot effort that 
we have brought before the Office of Management and Budget and 
hope to continue the discussions whether we can look at the pro- 
gram slightly differently? To try and sum it up for you, when you 
are developing research in science in a mission-oriented Agency 
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like our own, it is difficult to look at the results of that work in 
the absence of asking yourself how well has the program integrated 
the results that were to achieve the long-term outcomes? 

So what we are, in short, proposing is an integrated process, 
looking at both the program and the research arm to ask the ques- 
tion how is the Agency doing in getting the outcomes it wants with 
the research it is engaged in? 

Chairman Ehlers. As you know, we have had a number of dis- 
cussions about integration and I am very interested in strength- 
ening that effort within the EPA by increasing the scientific compo- 
nent of it. Mr. Posner. 

Mr. Posner. Thank you, Mr. Chairman. 

I want to make sure we are not saying that PART shouldn’t be 
applied to science programs. I think recognizing that there are par- 
ticularly unique challenges, given the long-term nature of how re- 
sults unfold and the uncertainties with science programs, means 
you have to be careful, and OMB does have particular questions for 
R&D. That may be one area that needs to be focused on. I want 
to add, thanks to the National Academy of Sciences, the National 
Science Foundation and others, there are developed criteria for 
doing this. 

One of the things that we typically find when we look at re- 
search, such highway research, is it the success story model, which 
is really not the way necessarily to go, where agencies reach for in- 
dividual good examples to justify their programs. But what we 
want to see and what has been developed in the National Science 
Foundation and others is a way for scientists to get peer-reviewed 
retrospective evaluations of portfolios of research. This is some- 
thing that has been developed in other agencies and it is the kind 
of thing that I think OMB has endorsed in their research criteria. 
The National Academies of Science really laid this out as a model. 

So when we are talking about having leading scientists review 
the results of science programs, that is the model of accountability 
that makes sense here. 

Chairman Ehlers. Well, thank you, and what we are specifically 
interested in is what specific measure should OMB use to evaluate 
EPA’s R&D programs, or for that matter, other R&D programs? So 
we will continue with Dr. Matanoski. 

Dr. Matanoski. I think our committee, the Science Advisory 
Board, in looking at the actual outcomes, suggested that the short- 
term outcome would fare much better in the PART review than 
would any long-term outcome. Now if you add on top of the long- 
term outcome something like ecological research, which is so com- 
plex, it has so many ramifications to it that when you go back and 
start looking at one basic piece of it and then carry it all the way 
to its completion at the end, which we don’t even do to be truth- 
ful — we don’t even know what is the best ecological end for most 
of what we are looking at, take that and try to produce an outcome 
with it becomes extremely difficult. 

It is easy to say I have to model something and I go in and I 
model that, like whether jellyfish are doing well in the Chesapeake 
Bay, and if you can do that, you can do that very well and you can 
do certain pieces of research and figure it out. But when you try 
to put the whole thing together, the entire fish population, the en- 
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tire growth population, the entire water systems of all of these, it 
becomes a very complex not only research arm, but a very complex 
outcome arm. And so the long-term outcome is you want something 
better in your coastal waters, for example. But all the pieces that 
have to go into the research to get there and then the application 
become too complicated to measure simply with a tool. 

I agree wholeheartedly that everybody has to be accountable for 
their money, in science as well as anywhere else. It is just the 
same as if we were in a business. But by the same token, when 
you get a complex program like that, it is very difficult, and our 
recommendation was that they begin to think about instead looking 
at a committee that looked at the tool along with an actual science 
review group, and then gradually, you could work aware, as you 
began to figure out what was the best way to do this. You could 
begin to work your way back and evaluate each piece. Right now, 
it is such a complex issue; it would be very difficult to do that. 

Chairman Ehlers. Dr. Denson. 

Dr. DENSON. Yes, sir. I am going to answer the question in two 
ways, from two perspectives. One is that from the perspective of 
the Committee, the NRC Committee. That Committee attempted to 
do guidelines at that time, those guidelines required the Committee 
to look at metrics of performance, quality and relevance of the pro- 
gram. And the Committee judged the programs on those bases. The 
metrics they used were both product and process metrics. The 
Committee discarded bibliometric types of analysis, quantitative 
types of analysis in evaluating the program, and instead concluded 
that complex research programs, and I use the word programs very 
carefully, require a different kind of evaluation. And that evalua- 
tion means bringing in a panel of experts to peer-review the pro- 
gram in all of its aspects. To bring in a panel, peer-review all of 
the aspects of the program. 

And as we point out in the report, we call this a level three or 
a level four evaluation. Level one evaluations, which look at indi- 
vidual grants, the progress of an individual research project can be 
graded quantitatively, and you can use bibliometric analysis for 
that. But the Committee concluded that that was inappropriate for 
a program as complex as the STAR program, which we consider to 
be a level three program. The ORD program in general would be 
a level four program, and you would need a much bigger, much 
broader panel of experts to review that. 

I am going to change for just a moment, sir, and bring out the 
fact that for four years, I served on the National Science Founda- 
tion’s Advisory Committee for the Engineering Directorate. I also 
served for two years on the National Science Foundation’s Direc- 
torate Advisory Committee for Environmental Research and Edu- 
cation. So I have a perspective of how it is done in one of the very 
best research agencies. And there, while they follow the GPRA and 
have the tools, ideas and people concern, there it is also difficult 
to do a metric on some of the programs until they have had a 
chance to fully develop. And my point about the STAR program is 
that it just needs a little bit more time, and I think that given that 
time and more results and having a review of a complex program 
by a peer-review panel would be the metric to use, and we say that 
in our report. 
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Chairman Ehlers. My time has expired. We will recognize the 
gentlelady from California, Ms. Lofgren. 

Termination of EPA’s Building Decontamination 
Research Program 

Ms. Lofgren. I wanted to explore a specific program and — be- 
cause I serve on the Homeland Security Committee as well, and I 
am on the Cybersecurity and Science Subcommittee, and we are ob- 
viously spending a lot of time in that area. And I was interested 
in the termination of the Building Decontamination Research Im- 
plementation Plan, and I was wondering, Dr. Gilman, it is my un- 
derstanding that last year, the National Academy of Sciences, at 
EPA’s request, reviewed that plan. They indicated, the National 
Academy did, that it was not possible really for EPA to get the job 
done in the timeframe remaining. I am interested in why EPA 
wants to terminate the program, rather than extending it and is 
that your judgment that building decontamination is now a lesser 
threat, or is that because this decision was made in conjunction 
with the Department of Homeland Security, Mr. Ridge, and is it 
going to be transferred to Homeland Security? Because we haven’t 
seen it surface yet in that department. 

Could you give me some report on 

Dr. Gilman. Sure. 

Ms. Lofgren. — that? 

Dr. Gilman. Let me start by giving you a little background of the 
research effort at EPA, which we are very proud of. We set out to 
establish a center there. It is something of a virtual center. We 
gave it a time, a sunset in creating the center, a three-year window 
to do the work. Why did we do that? We did that because we want- 
ed to really drive the folks involved in delivering products to first 
responders and to planners. So we created the center. Its mission 
was to work heavily through partnerships with Homeland Security, 
the Army, Air Force, Food and Drug Administration, all of the dif- 
ferent groups where we might find technologies and approaches to 
utilize in the mission areas for the EPA — they include water infra- 
structure, both drinking and sanitary. They include buildings, and 
it includes the use of tools for rapidly assessing risks for different 
factors. 

Those are the three areas the research center is focused on. We 
are in the process right now of working with those various agencies 
who view themselves as our customers to determine what we 
should do about that three-year sunset, whether there should be a 
further research effort within a center embedded elsewhere in the 
Agency. The decontamination program, that again was meant to be 
a three-year program. That is what the National Research Council 
reviewed for us. We are in the process now of trying to identify 
those things that were in the third year of our program that we 
would like to move up into the second year. We haven’t expended 
all the resources that have been appropriated for the first two 
years yet, so we do have the flexibility to change our priorities, and 
as I said, we are in consultation with Department of Homeland Se- 
curity and others now about what their desires are for our future 
work. 
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I think the products that we have turned out so far are quite im- 
portant and we are very, very proud of the productivity of that 
group to this point in time. 

Ms. Lofgren. It looks to me that the sun set on the thing. It is 
not really a work in progress, and it may be that we want to trans- 
fer the whole thing over to Homeland, but it is just I don’t think 
that has happened, and with so many things in the Homeland Se- 
curity area, not just environmental issues but where you have an 
activity going on by another agency and it never really gets picked 
up and it leaves the United States exposed and bare, I mean, in 
many respects, not just in the environmental arena, I am con- 
cerned that this is another one of them. 

My time is just about up, Mr. Chairman. I would just say that 
I think this has been a very useful hearing. One thing I think has 
become very clear in the questioning is that the mechanistic appli- 
cation of PART two science is not going to serve the country well, 
and I really appreciate your leadership, not only as Chairman, but 
as a scientist in helping this committee, and then later, I think the 
Congress to understand that that will not well serve our country 
and our future. So I yield back the time, and thank you once again. 

Peer Review and Evaluation of R&D Programs 

Chairman Ehlers. The gentlelady yields back her time. Let us 
continue with some of the previous discussion. There seems to be 
a general agreement among three of the panel that peer-review is 
the best way to review R&D programs. I just want to establish, Mr. 
Johnson, Dr. Gilman, do you agree that peer-review is the best 
method to review the science programs, the R&D programs? 

Mr. Johnson. Well, I will let, again, Dr. Gilman answer in a 
more detailed fashion, but I believe the PART has a very substan- 
tial role in the assessment of R&D programs. There is nothing 
mechanistic, not one thing that is mechanistic about the use of 
PART. Nothing happens automatically as a result of a PART score. 
I would like to just add one other comment — to elaborate on some- 
thing Dr. Gilman said about the building contamination research. 
The balance — the majority of the funds appropriated for this re- 
search in the last two years are un-obligated. They have not been 
spent or committed. There is already plenty of money available for 
this effort. Additional monies are not needed in the fiscal year 2005 
to continue this effort in an aggressive fashion. 

Chairman Ehlers. All right. Thank you. Dr. Gilman. 

Dr. Gilman. The pilot that we worked with the Inspector Gen- 
eral and the work that we would hope to do for the future does in- 
volve the use of peer-reviewers in evaluating both the research out- 
puts and the long-term outcomes. The PART process envisions the 
use of that kind of peer-review. The National Science Foundation 
has availed itself of that. Those are the kinds of things we hope to 
be discussing with the Office of Management and Budget for the 
future. 

Chairman Ehlers. Let me just comment. There are really two 
aspects, and I have seen these get interwoven here, but not clearly 
distinguished. It seems to me part of it is simply evaluating the 
science per se. Is the EPA doing good science? Is it judged by peers, 
it is really good work and should it be done? That is one aspect. 
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The other aspect is, in a sense, a management aspect. Does the re- 
search fit with the mission and the program of the EPA? And I 
think it is important to clearly distinguish those. Both of those are 
difficult to do, but they are of a different nature, and what is most 
difficult is to mesh the two. But isn’t that what you are trying to 
do, Dr. Gilman, with your work and Mr. Johnson, your primary in- 
terest, I believe, is does the science fit the program? Does this 
make sense for the government to do? Is that a fair statement? 

Mr. Johnson. Well, I would say more than that. Does it make 
sense for what our priorities are, is it promising fruit? Is it on 
track? Is the money being well-spent? 

Chairman Ehlers. Right. 

Mr. Johnson. Is it being spent in a responsible fashion? But I 
would also say that with regard to this — we are talking about gen- 
eralities, and we also need to talk about specifically this program, 
the Ecologic Research Program. I am not a scientist, but as I un- 
derstand it, Ecologic Research, the primary opportunity in this 
area of inquiry has to do with the opportunity for us as a country 
is better monitoring and better diagnosing. It is a pretty specific 
area. Therefore, the primary benefit or the opportunities to be pur- 
sued with research is to develop tools and databases to help in- 
formed decision-making. 

There is, for this kind of research, a much more specific kind of 
end product, if you will, which is something you don’t — it is not the 
right term for research, but it can — it is more purposeful than most 
research programs, and therefore, I think it lends itself more to 
PART assessment than basic research. 

Chairman Ehlers. Absolutely, and I don’t argue that. However, 
that is fairly low-level science, which makes it easy to analyze. I 
mean, monitoring and diagnosing is — well, I don’t want to get into 
Pasteur’s Quadrant here. Some of the scientists will know what 
this is. But that is in the lowest quadrant. The difficult part comes 
in, as one of the scientists mentioned here, synthesizing the infor- 
mation after you have monitored it, after you have diagnosed it, 
and then synthesizing it with other results from elsewhere. 

So you are looking at organism X and organism Y and organism 
Z and many others and seeing how they are interacting and what 
the net effect is, and you find some very surprising things there, 
which you don’t anticipate, some of which create greater environ- 
mental problems, some of which actually solve your problems. For 
example, depending — developing microbes that will consume toxic 
materials or oil. That is very basic research, but yet it is also re- 
lated to the program. If you are trying to clean up underground 
storage tanks, as we have been trying to do for over a decade, that 
is a very important issue, if you can clean those up with microbial 
action rather than digging it up, hauling it somewhere, treating it 
at a very high cost. 

So it all interrelates, and that is where my concern is. How are 
we evaluating how everything interrelates? And just without really 
knowing as much as I should about the evaluation program, it 
seems to me the PART program is ideal for what you just de- 
scribed, Mr. Johnson. But I don’t think it is — from what I have 
seen, it is not the appropriate instrument for the broader picture, 
and the question is can we all, working together, develop some- 
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thing that is more appropriate for the broader picture? We are not 
here to condemn EPA or OMB. We are all part of the same team. 
We are trying to help develop something that really does the job 
and does it in the best way for the country, and I hope everyone, 
both panelists and audience, understand this. Our goal is, as I 
often tell my people back home when I give a speech, I am here 
from the Federal Government, and I am here to help you 

Mr. Johnson. Yeah. 

Chairman Ehlers. — which usually brings a laugh. But that — I 
am serious about that. We really want to try to find out what the 
problems are and what we can do to help. Mr. Johnson. 

Mr. Johnson. But as Dr. Poser said, or Mr. Poser said, one thing 
I believe with all kinds of programs, including research and devel- 
opment, basic and applied, the PART is causing us to ask really 
good questions, and that is more important than the kind of an- 
swers that we are getting, but it is causing us questions that we 
weren’t asking before. 

Chairman Ehlers. And Mr. Johnson, I have no objection to that. 
I think that is an important function. The question is how those 
answers are evaluated and how they are used, and what the appro- 
priate use is. Did you have additional questions, Ms. Lofgren. 

Evaluating Short-term vs. Long-term Research 

Ms. Lofgren. I think I do, but I think we are going to be having 
a vote soon, and the only thing I would note is Dr. Matanoski, in 
your testimony, you describe research that is targeted to fill a re- 
search gap in the short run versus other types of research intended 
to provide an understanding of large systems in the long run, and 
I am just wondering if it is possible to have programs that have 
only short-term or long-term goals, and not both. And as I look at 
the PART questions themselves, it appears that really, you are 
going to get penalized, and you can’t get an approval if you have 
done that, and if your understanding is similar to mine. 

Dr. Matanoski. I think it is very similar to what you have said. 
You have hit it right on the nose. If you have something that is 
very simple that I am going to evaluate, an operative procedure or 
something, and I know what the long — what the outcome is going 
to be and they are going to come off the table alive or dead, that 
is my outcome. That is easy. And I can tell you how many oper- 
ations I do and how many are going to take place, you know, and 
who is going to do them. 

When you get into a program as complicated and integrated as 
ecological research, then you really begin to have to look at the 
independent parts, part of which was STAR. And so STAR was at 
the very basic end of it, and some of the more applied and the mon- 
itoring, which is fairly easy, you know. You put how many monitors 
out there. How much do they show us, and whether they have im- 
proved the quality of the water. But when you begin to look at the 
whole picture that is a very difficult thing to do. So what do we 
need to do? Separate it out so I have a product that I can say 
comes out next week, and if I get it out next week, great. I have 
just met my goal. And, you know, that is — if you get that simple, 
like putting a car on the line and coming out at the other end, then 
you can easily use a 
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Ms. Lofgren. Right. 

Dr. Matanoski. — very simple metric system. The more com- 
plicated you get, the more complicated any metric system would be- 
come, and right now, we are probably stuck with not having a very 
good metric system for a complicated problem. 

Ms. Lofgren. Right. In just looking at the questions, it is really 
biased against basic, complex research, it seems to me. I remember, 
Mr. Chairman, when I was first elected to the House and the late 
Congressman Steve Schiff was on the Science Committee and I was 
new to the Committee, and he actually gave an education to me 
and other new members about the value of basic research, and 
really its role in not just solving problems that are immediately be- 
fore us, but really in building the economy of the country. And I 
think we would be ill-advised to forget Steve’s lessons to us, and 
I yield back the balance of my time. 

Concluding Remarks 

Chairman Ehlers. I thank the Congresswoman for her com- 
ments, and I think we have kept you here long enough and tor- 
tured you enough, but I certainly want to thank the panelists for 
being here. You have been very, very helpful to me in helping me 
learn more about it from both sides of the issue, and I appreciate 
you being here. I would like to just summarize a few thoughts I 
have. So far, I have not heard any convincing reason why the 
STAR program should be cut so drastically, by all accounts. Not 
just what we have heard here, but other accounts I have heard. It 
is a well-run, competitive peer-reviewed program that produces 
high-quality research, and I don’t think the proposed reduction 
should take place, and so we will work with the appropriators on 
that and see what we can achieve. 

I also learned this morning the PART program was at least par- 
tially responsible for the funds being cut from Ecological and Pollu- 
tion Prevention Research Programs, but it doesn’t appear to me 
that PART is related at all directly to the cuts in the STAR pro- 
gram, and again, I don’t quite understand why the cut, but we will 
have further discussions. And finally, I believe that PART has been 
a valuable tool for OMB in a number of different areas, and I am 
just not yet convinced that it is the appropriate one as it is struc- 
tured to use for EPA and particularly I would be interested in how 
conclusions are drawn from the PART review and how they are 
used. It seems to me that the results of the PART program were 
not applied uniformly across the agency. That certain areas were 
cut more than they should have been, others were cut less than 
they should have been, if you look strictly at the PART review. And 
that is something else that we would like to investigate. Thank you 
again for being here. If there is no objection, the record will remain 
open for additional statements from the Members, and for answers 
to any follow-up questions the Subcommittee may ask of the panel- 
ists, and I hope you will be willing to answer in writing if any Sub- 
committee Members wish to send you communications. Without ob- 
jection, so ordered. 

Hearing is now adjourned. 

[Whereupon, at 11:30 a.m., the Subcommittee was adjourned.] 
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Answers to Post-Hearing Questions 

Responses by Clay Johnson, III, Deputy Director of Management, Office of Manage- 
ment and Budget 


Questions submitted by Democratic Members 

Ql. GAO’s review of the PART program indicated that PART and Government Per- 
formance and Results Act are not well integrated. GAO recommends these pro- 
grams be integrated to be more complementary. Does OMB intend to adopt this 
recommendation ? Will OMB alter PART to conform to GPRA or will agencies 
be instructed to alter their GPRA performance plans to conform to PART ? Since 
GPRA mandated an open process for the development of agency strategic plans 
and performance measures, what process will OMB use to include Congress and 
program stakeholders in the effort to conform PART and GPRA performance 
plans ? 

Al. Responding in part to GAO’s recommendation that PART and GPRA be better 
integrated, OMB clarified its PART guidance so agencies understood that PART is 
simply a tool to ensure GPRA is implemented as intended. The guidance now in- 
cludes the following: 

RELATIONSHIP OF THE PART TO THE GOVERNMENT PERFORM- 
ANCE AND RESULTS ACT (GPRA) 

The GPRA statute provides a framework under which agencies prepare strategic 
plans, performance plans, and, performance reports that set goals and report on 
the extent to which they are achieved. The PART is a systematic method of as- 
sessing the performance of program activities, focusing on their contribution to 
an agency’s achievement of its strategic and program performance goals. 

The PART strengthens and reinforces performance measurement under GPRA by 
encouraging careful development of performance measures according to the out- 
come-oriented standards of the law and by requiring that agency goals be appro- 
priately ambitious. Therefore, performance measures included in GPRA plans 
and reports and those developed or revised through the PART process must be 
consistent. 

The PART also extends the usefulness of performance information by defining 
programs that comprise activities about which management and budget deci- 
sions are made. As a matter of sound manage practice, agencies will integrate 
operational decisions with strategic and performance planning by: 

• improving performance measures over time through the PART review, 

• aligning budgets with programs, and 

• aligning programs and measures with GPRA goals. 

If agency goals are revised as a result of the PART process, agencies must engage 
in whatever consultation with stakeholders, including Congress, is required by 
GPRA. 

Q2. In cases where OMB and the Agency disagree about the performance measure 
used to evaluate a program with PART, how is the disagreement resolved ? 
Please provide documentation of the EPA and OMB positions on the PART eval- 
uation of the Ecosystem Research Program. Please include documentation of the 
negotiations that led to the final resolution of the differing positions and the 
awarding of the final PART evaluation score for this program. 

A2. OMB and agencies complete PART assessments together. We work hard to 
come to agreement on every answer to PART questions. Agencies can “appeal” OMB 
answers to PART questions to OMB policy officials, but this process is rarely used. 
The give and take between OMB and agencies in PART assessments is considered 
pre-decisional and therefore cannot be disclosed. However, OMB and EPA would be 
happy to discuss with you and/or your staff the process EPA and OMB went through 
to complete its PART assessments and, more importantly, what we are doing subse- 
quently to improve the programs we assessed. 

Q3. Under GPRA, program goals and performance matrices for programs are sup- 
posed to be adopted in consultation with Congress and outside stakeholders. 
However, it appears that PART-defined program goals and performance metrics 
are superseding those established under GPRA. Please provide a side-by-side 
comparison of the PART goals and performance metrics for the EPA programs 
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that have been assessed under PART in the 2004 and 2005 budget and the goals 
and performance metrics for these same programs developed under GPRA. 
Please provide an explanation of all instances where PART and GPRA program 
goals and performance metrics differ and OMB’s rationale for deviating from 
GPRA to develop PART. 

A3. As discussed above, the PART is simply a tool to ensure GPRA is implemented 
as intended. The PART is used to ensure, among other things, that program goals 
meet the high standards of GPRA. Where we find that existing goals do not meet 
those standards, agencies may be required to revise the goals associated with the 
programs being assessed. 

I am assured that EPA conducts extensive outreach to Congress and consults with 
its stakeholders when developing goals, objectives, and performance measures under 
GPRA. In its Strategic Plan and Annual Performance Plans, EPA seeks to use the 
highest-quality and most outcome-oriented goals and measures for which valid per- 
formance information is expected to be available. OMB and EPA consider EPA’s 
GPRA goals and measures to be the definitive set for Congress and the public to 
use to understand the agency’s budget in performance terms and to hold the agency 
accountable for results. 

The PART is not a substitute for GPRA. It is an assessment tool that informs 
management decisions, enhances organizational learning, and promotes effective 
strategies and program results. The PART is used to ensure, among other things, 
that program goals meet the high standards of GPRA. The PART process has been 
one of several drivers for improving the agency’s performance measures. Some of the 
new measures identified during a PART assessment will become GPRA measures. 
In many cases, newly identified measures are not ready immediately for inclusion 
in GPRA documents, since it will take some time for the agency to conduct appro- 
priate monitoring and data collection, to establish a baseline, and to set relevant 
targets. In these cases the agency develops specific plans, with milestones, for put- 
ting the necessary elements in place to use it as a GPRA measure. Each of the agen- 
cy’s program offices vets these measures with stakeholders, just as they do with any 
newly proposed measures. 

Some performance measures, notably efficiency measures, which are identified 
under the PART process, are more appropriately used as management tools. In 
other cases, PART assessments have been conducted on a relatively small set of 
agency activities that are not readily identifiable as a distinguishable unit in GPRA 
documents. In these cases, new measures identified under the PART process may 
complement GPRA measures, but they are not necessarily in GPRA documents. 

EPA included a discussion about the PART and performance measures in its most 
recent Annual Report, which we are including with this package (see pp. 12-13, 
Using Program Evaluation and Improving Environmental Indicators, Performance 
Measurement, and Data Quality and pp. 109-110, Support for Environmental Re- 
sults, Fiscal Year 2003 Annual Report, U.S. Environmental Protection Agency). Also 
included with this package are summaries of all EPA programs assessed with the 
PART. EPA, in full consultation with stakeholders, has increased the percentage of 
externally reported annual performance measures based on environmental outcomes 
(as opposed to activity-based outputs) from seven percent in the FY 1999 Annual 
Performance Plan to 36 percent in the FY 2004 Annual Performance Plan. 

Q4. OMB established two groups to consult with regarding the development of 
PART, the Performance Measurement Advisory Council and an Interagency Re- 
view Panel. What recommendations of these groups were adopted as revisions to 
PART for the 2005 budget presentation ? What is the current status of these two 
groups and what role will they play in the making adjustments to PART for the 
2006 budget ? Please provide documentation of the date of meetings held between 
OMB and the PMAC and the IRP, a list of persons attending these meetings, 
and a brief description of the issues discussed. 

A4. The Performance Measurement Advisory Council (PMAC) was established in 
June 2002 pursuant to the Federal Advisory Committee Act (FACA) to provide OMB 
with independent, expert advice relating to performance measurement. Pursuant to 
the statute, the PMAC’s duration was limited to nine months and all the meetings 
were public. Minutes from those meetings are available on the OMB website at 
http :/ lwww.whitehouse.govlomblbudintegrationlpmac-index.html. The PMAC’s 
recommendations are detailed in the meeting minutes. 

In fall 2002, OMB established an interagency panel to review a sample of draft 
PARTS completed the first year, judge their consistency with the PART guidance, 
and make recommendations for how consistency could be improved. The panel also 
reviewed a selected number of disagreements between agency and OMB staff on 



110 


how particular questions in the PART had been answered. The panel’s work review- 
ing draft PARTs is considered pre-decisional and therefore cannot be disclosed. In 
the two subsequent years of PART implementation, similar quality control exercises 
have been performed. In summer 2003, the National Academy of Public Administra- 
tion convened a working group to assist OMB with the PART consistency check. 
This summer, OMB conducted the consistency check on its own. We would be happy 
to brief you in greater detail on the substance of the groups’ work and the improve- 
ments that resulted from it. 
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Answers to Post-Hearing Questions 

Responses by Paul Gilman, Assistant Administrator for Research and Development, 
United States Environmental Protection Agency 

Questions submitted by Representative Zoe Lofgren 

Ql. In relation to the proposed rule on mercury emissions from coal-fired electric 
utilities (Fed. Reg. Vol. 69, No. 2: Friday, January 30, 2004; pp. 4651-4752): 

Qla. Provide a list and a brief description of the studies relied upon by EPA to de- 
velop the proposed mercury cap-and-trade program prior to the release of the 
proposed rule in January. Please indicate the author Is of these studies and in- 
dicate where EPA has made the studies available for examination by the gen- 
eral public and Members of Congress. 

Ala. EPA places all studies relied upon by EPA to develop the proposed and final 
rules in the docket. The Clean Air Mercury Rule docket can be found at http://dock- 
et.epa.gov/edkpub/index.jsp by searching for the docket ID OAR-2002-0056. 

Qlb. Administrator Leavitt has recently announced his intention to initiate and/or 
obtain additional analysis on the proposed mercury regulation. What addi- 
tional analyses will be undertaken ? What aspects of the proposed mercury cap- 
and-trade program for electric utilities will be examined by these new analyses? 
Will the analyses be undertaken by EPA’s Office of Air and Radiation or by 
EPA’s Office of Research and Development? What is the anticipated date of 
completion for the additional analyses? 

Alb. As Administrator Leavitt has clearly stated, EPA will make sure that we have 
all the analysis necessary to make the right decision about how to address mercury 
emissions from power plants. 

During the comment period on the proposal, EPA received a number of relevant 
analyses from various groups, including both industry and environmental groups. 
These analyses address many of the key issues faced by EPA and will help inform 
the Agency’s ongoing mercury work. They are available in EPA’s docket, and the 
Agency is seeking public comment on these analyses in its Notice of Data Avail- 
ability (NODA) released on November 30, 2004. The NODA is part of the EPA proc- 
ess toward delivering a final mercury rule by March 15, 2005. 

EPA received a number of modeling analyses from various groups, including both 
industry and environmental groups. In some cases, EPA and commenters modeled 
the same or similar policy scenarios, sometimes using the same model, but obtained 
substantially different results due to differences in the assumptions employed. In 
these cases, model-input assumptions can be better understood by comparing and 
contrasting the modeling performed. The NODA shares these analyses and seeks ad- 
ditional comment on the models and assumptions used. 

Administrator Mike Leavitt has outlined five guiding principles that provide con- 
text for additional inquiry and that narrows the focus of the Agency’s deliberations. 
The five principles will ensure that the final mercury rule: (I) concentrates on the 
need to protect children and pregnant women from the health impacts of mercury; 
(2) stimulates and encourages early adopters of new technology that can be ade- 
quately tested and widely deployed across the full fleet of U.S. power plants uti- 
lizing various coal types; (3) significantly reduces total emissions by leveraging the 
$50 billion investment that the Clean Air Interstate Rule (CAIR) will require; (4) 
considers the need to maintain America’s competitiveness; and (5) comprises one of 
many agency actions to reduce mercury emissions. 

Qlc. What study or combination of studies led to EPA’s assertion that the cap-and- 
trade mercury control program would reduce mercury emissions by 70 percent 
by 2015? What do EPA’s analyses show in regard to the anticipated change in 
the geographic distribution of U.S. mercury emissions from coal-fired electric 
utilities if the cap-and-trade mercury control program were adopted as com- 
pared to the reductions anticipated if the MACT mercury control program were 
adopted? 

Ale. The proposed mercury cap-and-trade program would place an emissions cap on 
mercury emissions from coal-fired electricity generating units. This cap would be im- 
plemented in two phases. The second phase of the program would begin in 2018, 
with a cap of 15 tons for emissions from these units. When this cap is fully imple- 
mented, emissions from affected units would be reduced by approximately 70 per- 
cent. The 15 ton cap represents a 70 percent reduction from the estimated 1999 
level of 48 tons of mercury emissions from coal-fired electricity generating units. 
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Once the 15 tons cap is fully implemented, emissions would not exceed that level, 
because the cap and trade mechanism places an absolute limit on allowable emis- 
sions. Because the cap is lowered in 2018, cumulative emissions reductions under 
the proposed cap-and-trade program would exceed cumulative emissions reductions 
under the proposed MACT. Furthermore, EPA analyses suggest that the proposed 
cap-and-trade program attains more reductions from larger and higher emitting fa- 
cilities than the proposed MACT. 

An important feature of the proposed mercury cap-and-trade program is the abil- 
ity for sources to bank emissions allowances. Sources can over-comply with the pro- 
gram in one period and bank remaining allowances for use in a later period. Bank- 
ing of allowances provides flexibility to sources, encourages earlier or greater reduc- 
tions than required, stimulates the market, and encourages efficiency. By encour- 
aging early reductions, banking provides early human health and environmental 
benefits relative to what would occur otherwise, though it results in extending the 
time until the cap is reached. We expect that sources will take advantage of the 
banking provision under the mercury cap-and-trade program by beginning to reduce 
emissions soon after the program takes effect. Allowing banking should not affect 
the cumulative mercury reductions achieved under the program. 


Questions submitted by Democratic Members 

Ql. EPA’s budget presentation includes program funding request changes for FY 
2005 relative to the Administration’s FY 2004 budget request. Please provide the 
changes for the following programs relative to the FY 2004 enacted appropria- 
tions for EPA including the funding level received in the FY 2004 enacted appro- 
priations law and the requested funding level for FY 2005. In cases where a de- 
crease in funding is proposed include a description of the activities proposed for 
termination. In cases where and increase in funding is proposed, include a de- 
scription of the new or expanded activities that will be undertaken. 

Qla. Office of Research and Development (ORD) research and development funding 
for mercury. 

Ala. Mercury Research, EPA’s Office of Research and Development. 


(Dollars in Millions) 


FY 2004 Enacted 

FY 2005 Request 

Increase/Decrease 

$6.9 

$5.2 

-$1.7 


EPA will maintain in-house research in this area, but plans to award five fewer 
Science to Achieve Results (STAR) grants for mercury research in FY 2005. 

Qlb. Office of Air and Radiation (OAR) research and development funding for mer- 
cury 

Alb. In the Science and Technology Appropriation in the FY 2004 Operating Plan, 
OAR has $361,700 for mercury activities; OAR has requested $361,700 in the FY 
2005 President’s Budget. This funding supports the regulatory and guidance pro- 
grams within OAR for mercury. 

Qlc. ORD research and development funding on fine particulates. 

Ale. Particulate Matter Research, EPA’s Office of Research and Development 


(Dollars in Millions) 


FY 2004 Enacted 

FY 2005 Request 

Increase/Decrease 

$58.6* 

$63.7 

$5.1 


‘Does not reflect S2M reduction to fund the Clean Automotive Technology Program as directed by Congress 
in FY 2004. 


The increase in resources will provide funding for a recently awarded 10-year, $30 
million Science to Achieve Results (STAR) grant to study the connection between 
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long-term exposure to air pollution and cardiovascular disease. The balance of the 
resources will support STAR research examining emerging biological mechanisms 
that can explain observed health effects associated with exposure to PM. 

The findings from these studies will assist policy-makers in understanding the 
public health consequences of long-term exposure to air pollutants, and will con- 
tribute important information for PM standard setting by increasing our under- 
standing of the biological plausibility of reported effects; the effects of PM and co- 
pollutants; who is most susceptible to these effects; and the effects of long-term ex- 
posure to PM. All these subjects are highlighted in the 2001 NRC Report IV on PM 
research priorities. 

Qld. Office of Air and Radiation (OAR) research and development funding for fine 
particulates. 

Aid. In the Science and Technology Appropriation in the FY 2004 Operating Plan, 
OAR has $33,009,800 for fine particulates activities; OAR has requested $42,762,500 
in the FY 20005 President’s Budget, This funding supports the regulatory and guid- 
ance programs within OAR, particularly the Mobile Source program. 

Qle. ORD research and development funding on ozone. 

Ale. Tropospheric Ozone Research, EPA’s Office of Research and Development 


(Dollars in Millions) 


FY 2004 Enacted 

FY 2005 Request 

Inc rease/Decrease 

$5.1 

$4.9 

-$0.2 


The decrease is due to a realignment of research support resources, and there is 
no programmatic impact to this research effort. 

Qlf. Offce of Air and Radiation (OAR) research and development funding for ozone 

Alf. In the Science and Technology Appropriation in the FY 2004 Operating Plan, 
OAR has $34,167,3000 for ozone activities; OAR has requested $36,224,400 in the 
FY 2005 President’s Budget. This funding supports the regulatory and guidance pro- 
grams within OAR, particularly the Mobile Source program. 

Q2. The FY 2005 Budget Request describes the Green Suppliers Network (GSN) ac- 
tivity within the pollution Prevention Program. How much funding does EPA in- 
tend to provide in FY 2005 to support this partnership program and meet the 
goals of expanding the GSN partnerships to include four additional industry 
sectors ? How much funding did EPA supply to support the GSN pilot effort in 
2002? What were the funding levels for this activity in 2003 and 2004? 

A2. The promise of the Green Suppliers Network (GSN) is best illustrated by the 
response to it by industry. The program is not yet a year old and already countless 
Original Equipment Manufacturers and their suppliers have joined the program. It 
is truly a win-win for all involved. The Original Equipment Manufacturers benefit 
because the program is designed to improve their supply chains both economically 
and environmentally. Small businesses benefit because they are being shown oppor- 
tunities to lean their operations and improve their environmental footprints, while 
at the same time achieving significant cost savings. The states benefit because the 
small businesses and their clients located within their jurisdictions remain competi- 
tive, leading to job retention and viability within the state’s economy. EPA and the 
environment benefit because the GSN outlines direct opportunities for environ- 
mental improvement from energy savings to waste minimization to the reduction in 
the use and emissions of toxic chemicals. 

When the program was launched, EPA did not expect the response from industry 
to be so strong and so immediate. Currently, The Green Suppliers Network is 
partnering with the Automotive Industry, the Aerospace Industry, the Office Fur- 
niture Industry and the Healthcare/Pharmaceutical Industry. However, EPA has 
been approached by several other industries which have shown an interest in par- 
ticipating in the program. The Transportation Industry represented initially by AM- 
TRAK is already engaged in a pilot program with GSN. The Farm Machinery Indus- 
try, represented by John Deere and others have written EPA with an interest in 
participating. The Trucks and Buses industries have also registered interest and 
most recently, the Retail Industry is opening a dialogue with EPA on this program. 
Additional resources are needed to ensure that the technical information the GSN 
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program offers is well packaged and effectively delivered to these new industries. 
Each industry that participates in this program is different and must first identify 
a list of suppliers within their supply chains which will benefit most from the pro- 
gram. Consequently, for each of these four industrial sectors, we could be addressing 
as many as 100 small to medium sized enterprises located across the United States. 
EPA, in partnership with NIST and the State Technical Assistance Providers, must 
make sure that they can deliver the GSN program effectively and consistently to 
all of these suppliers. This will require cross-training, more extensive communica- 
tion and planning and the tailoring of existing lean and clean tools to these new 
suppliers. Additional resources are needed to ensure that EPA can successfully 
merge lean technologies and environmental technical assistance for delivery to this 
extended supply chain. Moreover, GSN has attracted the attention of foreign govern- 
ments such as Canada and China, arid resources will be used to build a foundation 
for GSN implementation there. 

With additional resources one should expect to see measurable results from nu- 
merous small to medium sized enterprises and their Original Equipment Manufac- 
turers that are aggregated and ready to be reported in the near-tern. The results 
are collected from individual suppliers by the Manufacturing Extension Partnership 
program of NIST and aggregated for EPA. The results are measured through the 
tools that the NIST MEPs use in conducting an assessment. This includes, for exam- 
ple, the use of a tool called Value Stream Mapping which creates a material and 
information flow map of a product or process. This powerful tool allows companies 
to map the flow of products and information from order to cash as well as through 
the supply chain. Kaizen is another tool that is used to monitor and measure contin- 
uous improvement. These lean manufacturing tools and the environmental consider- 
ations that are being built into them through GSN are designed to measure im- 
provements as the process changes they recommend are implemented. Results meas- 
ured by these systems include pollution reductions, crater and energy savings, and 
company cost savings associated with those environmental benefits. These results 
will contribute to achievement of new results-oriented Pollution Prevention annual 
performance measures incorporated into EPA’s FY 2003 President’s Budget Request 
anal related Goal 5 Strategic Targets in EPA’s Strategic Plan for 2003-2008. 

Please see the table below for the requested funding levels. 


Fiscal 

Year 

Total 

Fund 

Ing 

Leve 

I 

2005 Presidents Budget 

$1,895,000.0 

2004 Enacted 

$485,000.6 

2003 Enacted 

$404,400.0 

2002 Enacted 

$213,100.0 


Q3. Do the goals, program definitions, and performance measures for research pro- 
grams that EPA has developed and included in its strategic plan align with the 
goals, program definitions, and performance measures associated with OMB’s 
PART analysis ? How will the discrepancies be addressed ? 

A3. The three EPA research programs that were reviewed through OMB’s Program 
Assessment Rating Tool (PART) analysis for the FY 2005 budget represent research 
programs as defined by EPA, each with its own multi-year research plan. These 
three programs (Particulate Matter Research, Ecosystems Protection Research, and 
Pollution Prevention Research) received PART scores of “Results Not Demonstrated” 
because the Agency and OMB were unable to reach agreement regarding appro- 
priate outcome-oriented goals and measures. For this reason, the PART worksheets 
for these programs note that goals and measures are under development. EPA is 
continuing to work with OMB to develop and identify appropriate outcomes for envi- 
ronmental research. 




